Plan: Performance testing extension

[timhall]

Goal: Add performance tests to output of tests (wouldn't necessarily be pass/fail, just info).

Possible designs:

' Custom matcher
With Specs.It("speed test as custom matcher")
    .Expect("Callback").RunMatcher "SpeedTest"

    ' Pros: Callback can be called multiple times (for average speed)
    '       Uses currently available methods 
    ' Cons: Not the most intuitive API
End With

' Method that takes SpecDefinition and Callback
SpeedTest(Specs.It("speed test with definition and callback"), "Callback", Args...)

' Pros: Simple API, callback can be called multiple times
' Cons: More intuitive, but still not perfect

' Most likely design
' ------------------
' Run, Start, and Finish
SpeedTest.Run Specs, "speed test with Run", "Callback", Args...
SpeedTest.Start Specs, "speed test with Start and Finish"
' ...
SpeedTest.Finish

[timhall]

(See VBA-Dictionary for example)

[robodude666]

I don't personally agree with performance benchmarking to be part of a testing framework; for that you have benchmarking suites like Benchmark.js. The role of TDD/BDD is to improve the software's design and raise code quality; not to ensure your code is fast.

I do agree that it would be handy to have test execution profiling and think it should be built into the SpecDefinition class so you can determine which tests are the slowest, however I'm finding it hard to find an elegant way of adding such functionality since the actual values are executed during Spec creation.

Maybe:

With Specs.It("should add an item")
    Set Dict = CreateDictionary(UseNative)

    .StartTimer

    Dict.Add "A", 123
    .Expect(Dict("A")).ToEqual 123

    .StopTimer
End With

which doesn't look that elegant... Or maybe:

With Specs.It("should add an item")

    Set Dict = CreateDictionary(UseNative)

    .Given(MyDict, "Add").WhenCalledWithArgs("A", 123).Expect(MyDict("A")).ToEqual 123

    ' Other examples could be:
    ' .Given("Callback").WhenCalled().Expect("...").To...
    ' .Given(MyObj, "Function").WhenCalled().Expect("...").To...
    ' and if you really need to:
    ' .Given("MySubroutine").WhenCalled().LogTime

End With

which is easier to read, but makes me feel icky for some reason.

[timhall]

@robodude666 Thanks for the comments! The second example is interesting, I'll ponder this more. I agree, this is out of scope for Excel-TDD, the goal was more to make sure that Excel-TDD provided some sort of foundation for enabling it.

[timhall]

It's been quite a while and I'm thinking this should be a separate library. There's no good way to get a pass or fail out of a speed test and I'd prefer to keep this library simpler.