Implement benchmarking support

[horenmar]

There were several requests for benchmarking support in Catch. This is not something that will be added to Catch Classic, but it might be added to Catch 2 at some point (probably not to the first release though).

[horenmar]

I will note that there is -d yes option, that makes Catch print out duration of individual tests (well, sections, which might make the interpretation of results a bit harder), but in no way is it a rigorous benchmark.

[nabijaczleweli]

I mean you're either not gonna do benchmarks (which is okay, this is a test framework), do them badly (even worse than not doing them) or do them right (in which case you could just use nonius instead).

IMO this is a classic Separation Of Concerns problem, and implementing benchmarks in a testing framework (possibly badly, since it's an extremely hard thing to get right) is a worse decision in every way than using nonius, which does them 100% right, is more fully-featured and is built around&for doing benchmarks.

[CD3]

I have no problem using a separate tool for bench marking. I have started using nonius because it supports a similar workflow to Catch, a header only library that I can include directly in my testing directory. I also like Celero because it allows you to setup a baseline to compare other benchmarks against, but it has to be compiled so there is some extra build configuration that has to happen.

But, what I really like about Catch is the SECTION feature. Not having to write a fixture class to do setup and teardown is great. I would really like a benchmark tool that supported sections. So, should we ask the nonius maintainer to add support for Catch-style fixtures?

[philsquared]

I've since used nonius, too (thanks for the recommendation, @nabijaczleweli). It's pretty good, but I also have a wishlist. I'm still considering supporting benchmarking in Catch (probably as a separate, optional, header) - but certainly talk to R. Martinho over on nonius.

[CD3]

Well, I've been using nonius for a bit now, and I think this would be a nice feature to add to Catch. Specifically, I find myself wanting to add CHECKs after running a benchmark to make sure the benchmark is valid (meaning I didn't typo a bug into it). Of course, I can write a seprate Catch unit test, but then I'm writing two copies of the same tests.

I really like nonius's approach to timing. It automatically determines how many times to run the test to get a "good" measurment, and performs a statistical analysis on the timing measurments. I think I heard @philsquared say that he did something similar on a different project.

So, my idea is to add a MEASURE macro that acts like a SECTION but wraps its body in something like nonius's chronometer.measure() function. This would allow benchmarks to be embedded directly into tests, so that assertions could be made about the state of the test.

[Glavnokoman]

Just came across this Catch2/blob/master/projects/SelfTest/UsageTests/Benchmark.tests.cpp. It seems Catch2 does support simple benchmarking. Not very precise though but just as convenient as the rest of it.

[horenmar]

The just released version in fact contains slightly adapted Nonius code.