psf / pyperf

Toolkit to run Python benchmarks
http://pyperf.readthedocs.io/
MIT License
799 stars 78 forks source link

Collect specialization statistics from running benchmarks #140

Closed mdboom closed 1 year ago

mdboom commented 2 years ago

This will collect statistics on CPython builds with --enable-pyperf, and only during the run of the benchmarks themselves.

Cc @markshannon

mdboom commented 2 years ago

The statistics are collected in the specializing optimizer in the interpreter loop in CPython itself when built with --enable-pystats. The statistics are dumped in separate files at process shutdown in /tmp/py_stats (which is how this works with multiple processes). You can see some example output of the summarize_stats.py script here: https://github.com/faster-cpython/ideas/blob/main/stats.md

mdboom commented 1 year ago

Anything else you'd like to see on this, @vstinner?

mdboom commented 1 year ago

I'm converting this to a draft -- on full A/A testing, it turns out the results from this a quite non-deterministic. I think the noise is coming from the warmup / calibration stages, which don't always run the same number of times. I think by excluding these runs, we'll get more stable results, but checking that over here first.

mdboom commented 1 year ago

This is ready for review. With this change, we get consistent results running the pyperformance benchmark suite multiple times. See this prototype.