Open robodude666 opened 6 years ago
I'm heading towards method-per-test too, I've been bitten by variables / errors from other tests causing issues too. I have some plans for how to build the process into vba-test and will definitely take a deep look at your suggestions. I'll get back to you on this
Good plan fellas. I got something similar working in a different way last December starting at 4c34095 and improved through 0d8b936
basically, I added a collection of specsuites to the specsuite class, allowing nesting. it's not explicitly one test per method but you could do that if you wanted to.
it made reporting of large numbers of tests more organized in the workbook reporter as well by allowing for collapsible, nested reports.
how to structure the code is in the Readme
example WorkbookReporter output
food for thought at least I hope. 😄
@timhall good to hear! Looking forward to it :+1:
@connerk I originally tried something similar but the issue I have with it is you need to remember to add the "sub" test suite. The benefit of discovery (as outlined above) is all you need to do is add a public sub and the test will automatically run.
any consideration on using comment markers? Rubber duck calls them Annotations NUnit calls them Attributes
@timhall btw, what were you thinking of?
@connerk The way RubberDuck handles Annotations via an ANTLR grammer makes it really easy on the implementation. However, if we do it from VBA you can't really do that very well. In VBA you can very easily iterate through the CodeModule
fairly efficiently and get the name of the procedure (via ProcOfLine
).
It is technically possible to use the same method and then parse the comments before the beginning of the procedure, but I donno.
I do like the idea through of supporting skipping tests or marking them as expecting to fail, but I don't like the idea of needing to add extra stuff to make a test run. Perhaps a combination of both? Using signature to detect "TestMethod" and annotations to mark tests for skip/fail?
Generally, I see vba-test as a low-level testing library that can provide the backing for various code generation, annotation, and other approaches. With #27 I think both approaches presented here are doable. I imagine @robodude666 example like the following:
Public Sub RunTests()
Dim Suite As New TestSuite
Suite.Name = "Project Name"
Dim Reporter As New WorkbookReporter
Reporter.ConnectTo ThisWorkbook.Worksheets("TestRunner")
Reporter.ListenTo Suite
Dim DebugReporter As New ImmediateReporter
DebugReporter.ListenTo Suite
' From external library
LoadTestsFromModule Suite.Group("Module Name"), "ModuleName"
LoadTestsFromClass Suite.Group("Class Name"), "ClassName"
End Sub
' Example
Public Sub LoadTestsFromModule(Suite As TestSuite, Name As String)
' Load CodeModule for Name
' Iterate through module looking for ' #[test] attribute
' -> Get test name (and modifiers)
On Error Resume Next
For Each TestName In FoundTests
Set Test = Suite.Test(TestName)
Application.Run Name & "." & TestName, Test
Test.NotError
Err.Clear
Next TestName
End Sub
Public Sub LoadTestsFromClass(Suite As TestSuite, Name As String)
' Similar to above, but with CallByName
End Sub
The point being that LoadTestsFromModule
can happen outside of vba-test. With that said I plan on integrating test generation into vba-blocks (once I get it working and released). That will generate and inject the test runner code based on ' #[test]
"attributes". This would generate output like the following:
Public Sub RunTests()
Dim Suite As New TestSuite
Suite.Name = "Project Name"
Dim Context As New Tests_Context ' <- Generated by vba-blocks test
With Suite.Group("TestCase")
Context.TestCase_ListenTo .Self
Context.TestCase_ShouldDoA .Self.Test("should do a")
Context.TestCase_ShouldDoB .Self.Test("should do b")
End With
With Suite.Group("TestSuite")
Context.TestSuite_ListenTo .Self
Context.TestSuite_ShouldDoC .Self.Test("should do c")
Context.TestSuite_ShouldDoD .Self.Test("should do d")
End With
End Sub
This would allow writing tests in modules or classes like the following:
' #[test.before_each]
Public Sub BeforeEach(Test As TestCase)
' ...
End Function
' #[test]
Public Sub ShouldAddTwoNumbers(Test As TestCase)
' ...
End Function
' #[test.skip]
Public Sub SkippedTest(Test As TestCase)
End Function
' #[test("override test name").only]
Public Sub ShouldAddAnyNumberOfNumbers(Test As TestCase)
' ...
End Function
' #[test.group]
Public Sub SubGroup(Suite As TestSuite)
With Suite.Test("...")
' ...
End With
End Sub
With all that said, I think vba-test will stay somewhat minimal, with either separate packages finding and running tests or via an outside tool like vba-blocks generating test runners
I really enjoy
vba-tdd
for it's simplicity, however, I find it limiting when developing large test suites. The fact that everything must be in a single function can either make for very long messy suites or create a ton of small suites. Splitting into sub-suites is also difficult due to VBA's module name size limitation. I've ran into theProcedure too large
error several times so far too.It would be helpful to support running test suite classes where each method is its own test; this will help with a number of things as I'll discuss below.
As an example, the
AddTests
example could be written as:Note: Technically error handling within the test subroutine would be optional; see below.
Because VBA does not support programmatically creating classes the user would have to specify a "factory method" to the reporter:
The reporter would be able to: 1) Detect the suite's class name from the return type of the factory. 1) Detect all Public Subroutines within the class. 1) For each subroutine detected: 1) Call the factory method to create a new instance. 1) The name of each test can be extrapolated from the PascalCase/SnakeCase name of the test. e.g.
ShouldAddTwoNumbers
andshould_add_two_numbers
would both convert to "should add two numbers" 1) Create a newTestCase
for theTestSuite
based on the above name. 1) Call each subroutine while passing in theTestCase
instance.Note: To support class reuse
SetUp
/TearDown
andSetUpSuite
/TearDownSuite
subroutines could be used.This has several benefits:
1) Allows for large test suites to be broken out into individual test subroutines. 1) Ensures each test runs in a fresh context. You don't have to worry reusing variables that might affect the outcome of your test. 1) Better error handling within the test itself. 1) Provides global error handling for unhandled errors (when the Reporter calls CallByName it can detect if an error occured). 1) Not having to worry about updating
NumSuites
. 1) Provide the capability of timing how long each test and suite takes to run. 1) Potentially support rerunning only failing suites/tests.In order to make
TestSuite.Create
work thePredeclaredId
would have to be set to True.It may also be helpful to add a generic runner class instead of using the WorkbookReporter/ImmediateReporter directly:
Final closing notes: I bring this suggestion up because I really like the all-included aspect of
vba-test
. While there are more powerful solutions available for vba testing (like RubberDuck/SimplyVBUnit (with modification)), they require external installation which in my particular application is not a feasible option. I strongly feel such an enhancement tovba-test
will make it much more flexible.Thoughts? 😄