search

twisted / twisted

Event-driven networking engine written in Python.
https://www.twisted.org
Other
5.49k stars 1.15k forks source link

Expand benchmarks to run on PyPy as well #12174

Open itamarst opened 1 month ago

itamarst commented 1 month ago

Currently the codspeed.io run on CPython only, but PyPy is also important.

Running benchmarks on PyPy requires a slightly different runtime approach, insofar as it has a JIT (and future CPython may add JIT too, of course). In particular, my impression is that codspeed will only run the code once, and measure instruction count. This is fine for CPython, but for a JIT it will give misleading results. So the benchmarking framework should run code multiple times if it's on PyPy.

This can be done with a custom pytest fixture.

Separately, there's the issue of how to communicate different runs to codspeed, which I will also look into.

itamarst commented 1 month ago

@p12tic FYI.

itamarst commented 1 month ago

OK, as mentioned at https://github.com/CodSpeedHQ/pytest-codspeed/issues/33 PyPy benchmarks are a Problem. Maybe not a problem for Twisted though, so that may not be a blocker, will have to do some more investigation.