Open itamarst opened 2 months ago
The worry is that the JIT compilation will ... "over-optimize" the code. See e.g. this explanation of why this Rust benchmark framework requires the use of the black_box()
function: https://bheisler.github.io/criterion.rs/book/getting_started.html#details
I will do some investigation and report back.
This is definitely going to be a problem (and likely for pytest-benchmark
as well). There's the equivalent of block_box
in PyPy (pypyjit.residual_call()
) but ... it may also require intervention by benchmark authors.
Further discussion with PyPy authors suggests that over-optimization of code (where you end up benchmarking code that won't match real world code) is not really solvable by the framework. It will likely require changes to the framework to make it easier for benchmark authors, documentation change, and benchmarks to be written in a particular way.
If CPython gets a JIT, this will likely become an even bigger issue.
I am therefore going to do some more research and then make suggestions that will presumably require pytest-benchmark changes too. Will report back.
Thanks a lot for the investigation. I totally agree with your Point of View, and it's already quite painful in compiled languages. When that happens, we will measure both the cold and JIT-optimized functions since the data will make sense to compare.
Definitely, this optimization topic will be a big problem. One way to "solve" it would be to ignore and disable the JIT, but this won't match real-world data.
You also mentioned warmup to make the JIT optimize the function; this might be hard. We ran some experiments with V8 but didn't find a definitive way to ensure the optimization of the whole execution path. Is there anything to force optimization with Pypy? I'm not aware if such a thing is also planned with CPython.
With PyPy you run the function ~3000 times and that triggers JIT compilation.
PyPy uses a JIT, so the initial run of a function may be misleading: after some number of runs the JIT kicks in, and then it will run much faster. CPython is working on a JIT too, so this may be relevant to future CPython releases as well (possibly even 3.13).
The code would likely look like: