Open travishathaway opened 9 months ago
This would be lovely but first we need to assess how stable the benchmarks are or, more accurately, how they are run.
If the plan is to run it on public CI, the variance of the runtime conditions makes it, essentially, a very expensive false positive generator :D This is the blog post I wrote about it. This other one is also useful to get an idea on how noisy it can get.
CI is only noisy due to run time though [^1], so if we can measure performance via other metrics (e.g. number of operations), then we should be ok. This blog post explains how cachegrind
can be used to do so, but it assumes the benchmarks are merely CPU bound. We need to check how IO is accounted for in the tests.
[^1]: GHA runners may use different CPUs:
Dv2-series run on the 3rd Generation Intel® Xeon® Platinum 8370C (Ice Lake), Intel® Xeon® Platinum 8272CL (Cascade Lake), Intel® Xeon® 8171M 2.1GHz (Skylake), Intel® Xeon® E5-2673 v4 2.3 GHz (Broadwell), or the Intel® Xeon® E5-2673 v3 2.4 GHz (Haswell) processors with the Intel Turbo Boost Technology 2.0.
Checklist
Summary
This project has not been touched for some time; therefore, the goal of this epic is to revisit the work done here, possibly add to or remove some benchmarks, and get it ready for publishing elsewhere (e.g. conda.org of the conda documentation).
The work will be organized into two separate groups:
Linked Issues & PRs