ewinston
commented
2 years ago Since one often is looking to compare specific metrics, and the metrics might vary between benchmarks, it might be useful to use a lightweight exploratory data analysis (EDA) tool which allows one to quickly select, filter, and plot data of interest.
Some possible tools are dtale, pandasgui and a few others.
With dtale you can easily create summaries like averaging over algorithm and tool,
Here is a look at min, max and mean timings after filtering for tool=qiskit, "QSE" in name,
The table-based output from req-queen is currently a bit unwieldy, especially as the number of benchmarks, compilers, versions and platforms grow. We should find a way to quickly visualize and compare results generated by red-queen across the above dimensions. There are many prior examples of this being done for other languages or other compilers, e.g.:
https://perf.rust-lang.org/index.html https://benchmarksgame-team.pages.debian.net/benchmarksgame/fastest/rust-gpp.html
Also, the MetriQ platform has a section focused on quantum compilers ( https://metriq.info/Task/57 ) which also allows for comparisons across compilers and metrics, to which we could submit red-queen results.