search

python / pyperformance

Python Performance Benchmark Suite
http://pyperformance.readthedocs.io/
MIT License
869 stars 175 forks source link

Upgrade to pyperf 2.6.3 #330

Closed corona10 closed 8 months ago

corona10 commented 8 months ago

See: https://github.com/python/cpython/issues/116024

Except for coverge.py related benchmarks, pyperformance is runnable from a free-threaded build. cc @colesbury

corona10 commented 8 months ago

I am going to release 1.11.0 after this PR is merged.

corona10 commented 8 months ago 
[ 1/87] 2to3...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_2to3/run_benchmark.py --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmp35jve2ds --inherit-environ PYPERFORMANCE_RUNID
Processing /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_2to3/vendor
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: lib2to3
  Building wheel for lib2to3 (pyproject.toml) ... done
  Created wheel for lib2to3: filename=lib2to3-3.12-py3-none-any.whl size=316110 sha256=8aabd67bca88aff2feca3cb24911afab35123161c09a24c59bff14abd41ea8ff
  Stored in directory: /private/var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/pip-ephem-wheel-cache-deea6cdm/wheels/fa/2d/65/a45c8b749ba13b16f2223aac84f488631d07aeba421f4116ca
Successfully built lib2to3
Installing collected packages: lib2to3
Successfully installed lib2to3-3.12
.....................
2to3: Mean +- std dev: 291 ms +- 9 ms
[ 2/87] async_generators...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_generators/run_benchmark.py --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmp5u48eh0y --inherit-environ PYPERFORMANCE_RUNID
.....................
async_generators: Mean +- std dev: 435 ms +- 10 ms
[ 3/87] async_tree...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_tree/run_benchmark.py none --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmpie6ecfor --inherit-environ PYPERFORMANCE_RUNID
.....................
async_tree_none: Mean +- std dev: 296 ms +- 10 ms
[ 4/87] async_tree_cpu_io_mixed...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_tree/run_benchmark.py cpu_io_mixed --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmpplqkqqn9 --inherit-environ PYPERFORMANCE_RUNID
.....................
async_tree_cpu_io_mixed: Mean +- std dev: 581 ms +- 12 ms
[ 5/87] async_tree_cpu_io_mixed_tg...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_tree/run_benchmark.py cpu_io_mixed --task-groups --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmpiuxm_hrj --inherit-environ PYPERFORMANCE_RUNID
.....................
async_tree_cpu_io_mixed_tg: Mean +- std dev: 558 ms +- 17 ms
[ 6/87] async_tree_eager...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_tree/run_benchmark.py eager --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmpn43hsm0w --inherit-environ PYPERFORMANCE_RUNID
.....................
async_tree_eager: Mean +- std dev: 121 ms +- 2 ms
[ 7/87] async_tree_eager_cpu_io_mixed...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_tree/run_benchmark.py eager_cpu_io_mixed --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmpeg2x1dd3 --inherit-environ PYPERFORMANCE_RUNID
.....................
async_tree_eager_cpu_io_mixed: Mean +- std dev: 436 ms +- 11 ms
[ 8/87] async_tree_eager_cpu_io_mixed_tg...
# /Users/user/oss/foo/venv/cpython3.13-b3c68e7e7579-compat-a211859439da/bin/python -u /Users/user/oss/cpython/.venv/lib/python3.13/site-packages/pyperformance/data-files/benchmarks/bm_async_tree/run_benchmark.py eager_cpu_io_mixed --task-groups --output /var/folders/xf/8mmvc8hx4ld687g2bp49zmkw0000gn/T/tmpsdsn1xsw --inherit-environ PYPERFORMANCE_RUNID
.....................
corona10 commented 8 months ago

@mdboom @ericsnowcurrently Would you like to take a look?

corona10 commented 8 months ago

The only uncertainty I have is if this change could skew benchmark results in any meaningful way. I

Thank you, Eric; from pyperf 2.6.2 to 2.6.3, there are only 2 changes.

So, the current default build-based benchmark will not be affected by these changes. And I believe that an official benchmark comparison for free-threading should be handled when the free-threading build can execute pyperf without opt-out changes.

ericsnowcurrently commented 8 months ago

Sound good. I'm satisfied. Thanks for explaining.