Closed austin3dickey closed 7 months ago
Totals | |
---|---|
Change from base Build 7450545287: | -0.04% |
Covered Lines: | 1544 |
Relevant Lines: | 1772 |
That run was posted to Conbench, and as expected each result has 20 repetitions. That's exciting.
The bad news: the cpp-micro benchmarks went from taking 78 minutes to 209 minutes. This would add fairly significant load to the machines. I'll try bringing the repetitions down to 10 and see if the total time decreases.
Yep, it decreased to 117 minutes. Almost a proportional decrease!
If this is a linear relationship, then decreasing to 6 repetitions should end up with a total time around 80 minutes again. I'll do that, merge the PR, and monitor.
This PR tells Archery to run each cpp microbenchmark executable with the instruction to do ~20~ 6 reps, where each rep runs as many iterations as it can in 0.05 seconds (or a bit more if it's statistically necessary). This should smooth out the data since we now alert on the "best rep" instead of the mean, resulting in fewer false positives and less outlier weight.