voltrondata-labs / benchmarks

Language-independent Continuous Benchmarking (CB) for Apache Arrow
MIT License
10 stars 11 forks source link

Bump cpp-micro repetitions to 6 #157

Closed austin3dickey closed 7 months ago

austin3dickey commented 7 months ago

This PR tells Archery to run each cpp microbenchmark executable with the instruction to do ~20~ 6 reps, where each rep runs as many iterations as it can in 0.05 seconds (or a bit more if it's statistically necessary). This should smooth out the data since we now alert on the "best rep" instead of the mean, resulting in fewer false positives and less outlier weight.

coveralls commented 7 months ago

Pull Request Test Coverage Report for Build 7669885589


Totals Coverage Status
Change from base Build 7450545287: -0.04%
Covered Lines: 1544
Relevant Lines: 1772

💛 - Coveralls
austin3dickey commented 7 months ago

testing here: https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-arm64-m6g-linux-compute/builds/4340

austin3dickey commented 7 months ago

That run was posted to Conbench, and as expected each result has 20 repetitions. That's exciting.

The bad news: the cpp-micro benchmarks went from taking 78 minutes to 209 minutes. This would add fairly significant load to the machines. I'll try bringing the repetitions down to 10 and see if the total time decreases.

austin3dickey commented 7 months ago

Yep, it decreased to 117 minutes. Almost a proportional decrease!

If this is a linear relationship, then decreasing to 6 repetitions should end up with a total time around 80 minutes again. I'll do that, merge the PR, and monitor.