mquinson
commented
6 years ago I've investigated this a bit today, and here are some thoughts.
On one side, google/benchmark is rather heavy weighted, and it seems to be somehow tight to google/test which seem to be a replacement of Boost.test that we use for now. We could decide to move away of Boost.test, as we don't have that much tests using this framework yet and as it's not present on older build systems anyway. But I'm not sure that google/test is lighter and nicer to use than Boost.test in any way.
On the other side, I've found a nice lightweighted competitor of google/benchmark: Nonius. It's a header-only library depending on boost components for the timing and the maths, so it'd be easier to integrate it into our framework. But I'm not sure of how robust it is against noisy computation conditions.
The feature I'm missing in Nonius compared to Google.benchmark is the ability to compute the Big O complexity of a test. This would be great to abstract away the changing computation settings that we get on our highly virtualized CI infrastructure. It seems uneasy to implement on top of Nonius, as it does not take care of the test input size by itself.
I'll have to dig a bit further, but wanted to post what I've found already.
We already have some benchmarking code in SimGrid, such as teshsuite/surf/maxmin_bench or teshsuite/xbt/parmap_bench or teshsuite/simdag/evaluate-parse-time. But in practice, they are only used to test the correction of the feature, not to benchmark things.
I think that we should really do the next move and integrate something like https://github.com/google/benchmark to our test settings. https://testing.googleblog.com/2007/10/performance-testing.html is interesting for more information.
Hopefully, that would integrate nicely in our jenkins settings, so that we get another family of nice graphs.
What would you guys think of it? Any better possible solution to use?