It's not clear whether it makes sense to include the "warmup" benchmark runs in the timings. On the one hand, things out of our control like disk cache etc. may still be settling out and it would be good to ignore that. On the other hand, it may hide the time required to manage and perform optimizations, which for some benchmarks may occur during warmup.
It's easy to run an experiment so see the effect of including it in the measurement, because pyperf has always included the warmup times in the output (but doesn't use them when computing a timing comparison).
Using a recent CPython main (9da98c) against 3.10.4 and comparing the HPT at 99th percentile:
Run type
Without warmups
With warmups
Tier 2 optimizer on
1.12x faster
1.11x faster
Tier 2 optimizer off
1.25x faster
1.25x faster
So, this doesn't make a measurable difference when the Tier 2 optimizer is off. As one would expect, including warmups with the Tier 2 interpreter on makes the numbers slightly more pessimistic.
Personally, I think we shouldn't mess with this right now, but I would like feedback from the rest of the team.
It's not clear whether it makes sense to include the "warmup" benchmark runs in the timings. On the one hand, things out of our control like disk cache etc. may still be settling out and it would be good to ignore that. On the other hand, it may hide the time required to manage and perform optimizations, which for some benchmarks may occur during warmup.
It's easy to run an experiment so see the effect of including it in the measurement, because pyperf has always included the warmup times in the output (but doesn't use them when computing a timing comparison).
Using a recent CPython main (9da98c) against 3.10.4 and comparing the HPT at 99th percentile:
So, this doesn't make a measurable difference when the Tier 2 optimizer is off. As one would expect, including warmups with the Tier 2 interpreter on makes the numbers slightly more pessimistic.
Personally, I think we shouldn't mess with this right now, but I would like feedback from the rest of the team.