Open jonstewart opened 1 year ago
Same usecase here. I am in the process of choosing the implementation language. Which mean I am running microbenchmarks available on those languages and comparing the results with rust implementation. I would like to plug-in the results to create the report.
I am considering using --message-format=json
to get data-points from rust implementation and do the analysis myself. But this would mean re-implementing a lot of things from scratch.
Criterion is great! I work across Python, C++, and Rust. One thing I have wanted in a benchmarking tool is the ability to feed in raw benchmark data from outside the framework. For example, in C++, the Catch2 unit test framework has some microbenchmarking facilities that will run benchmark functions for a number of iterations in order to get a good sample.
What I would love to be able to do is feed in the data from Catch2 benchmarks to Criterion, in order to get the commit-aware analysis, reports, and plots. It would also be nice to do similarly with acceptance test-level benchmarks involving complicated cloud data processing pipelines (where results may be measured in minutes) — things that are often scripted out, but without the analysis and reporting support that Criterion provides.
Is this possible with some jiggery-pokery in Criterion, or would ever be possible? Even something hacky would beat rolling my own solutions for other languages.