search

faster-cpython / ideas

1.7k stars 49 forks source link

Measure also PyPy and GraalPy in benchmarking-public ? #705

Open paugier opened 1 week ago

paugier commented 1 week ago

I realize that this project is about CPython, but it seems to me that it would be useful for the Python community to also include in https://github.com/faster-cpython/benchmarking-public few performance measurements of stable releases of PyPy and GraalPy.

Currently it's quite difficult to compare the numbers given by the different teams. PyPy and GraalPy both state that they are approximately 4 times faster than CPython. But it would be nice to get numbers measured using your measurement method. I guess that PyPy and GraalPy developers would be available to help for the setup if needed so it would not add too much work for the Faster CPython team.

For the Faster CPython project, it would give good ideas of what can be gained.

And for the scientific/data Python community it would be a good indication of how interesting it would be to go towards an ecosystem more gentle with alternative interpreters , using in particular C API like HPy (mentioned also in https://github.com/faster-cpython/ideas/blob/main/3.14/README.md).

So, my question is: would it be possible to also include few measurements using PyPy and GraalPy in https://github.com/faster-cpython/benchmarking-public ?

mdboom commented 5 days ago

I like the suggestion, and I would like to see it. I think speed.python.org had this as a goal at one time. However, it's a pretty expensive proposition, and I think we'd need more resources in order to be able to support it.

I feel very strongly about keeping the tools to reproduce these results open source and as a point of collaboration. Specifically, pyperf (measurement and hardware tuning fundamentals), pyperformance (a set of benchmarks and benchmark-running harness), python-macrobenchmarks (additional macrobenchmarks) and bench-runner (infrastructure for running all of the above on Github Actions). From this, alternative implementations can see exactly what and how we are measuring and reproduce themselves. I see this as sort of the principles of "doing good science".

markshannon commented 4 days ago

The best available comparison of GraalPython, PyPy and CPython from a performance perspective is https://onlinelibrary.wiley.com/doi/10.1002/spe.3267. I don't know if there is a non-paywalled version anywhere.