Faster CPython Benchmark Infrastructure
Results
Here are some recent and important revisions. 👉 Complete list of results.
Most recent pystats on main (dbd2379)
linux x86_64 (linux)
| date | fork/ref | hash/flags | vs. 3.12.6: | vs. 3.13.0rc2: | vs. base: |
|---|---|---|---|---|---|
| 2024-12-03 | python/bfb0788bfcaab7474c1b | bfb0788 | 1.116x ↑ 📄📈 |
1.070x ↑ 📄📈 |
|
| 2024-12-03 | python/bfb0788bfcaab7474c1b | bfb0788 (NOGIL) | 1.204x ↓ 📄📈 |
1.231x ↓ 📄📈 |
1.277x ↓ 📄📈🧠 |
| 2024-12-01 | python/7ea523f47cdb4cf512a1 | 7ea523f (NOGIL) | 1.199x ↓ 📄📈 |
1.230x ↓ 📄📈 |
1.247x ↓ 📄📈🧠 |
| 2024-12-01 | python/7ea523f47cdb4cf512a1 | 7ea523f | 1.075x ↑ 📄📈 |
1.033x ↑ 📄📈 |
|
| 2024-11-30 | python/328187cc4fcdd578db42 | 328187c | 1.069x ↑ 📄📈 |
1.025x ↑ 📄📈 |
|
| 2024-11-30 | python/328187cc4fcdd578db42 | 328187c (NOGIL) | 1.209x ↓ 📄📈 |
1.239x ↓ 📄📈 |
1.251x ↓ 📄📈🧠 |
| 2024-11-29 | python/38264a060a8178d58046 | 38264a0 (NOGIL) | 1.196x ↓ 📄📈 |
1.227x ↓ 📄📈 |
1.242x ↓ 📄📈🧠 |
| 2024-11-29 | python/38264a060a8178d58046 | 38264a0 | 1.070x ↑ 📄📈 |
1.024x ↑ 📄📈 |
linux x86_64 (vultr)
| date | fork/ref | hash/flags | vs. 3.12.6: | vs. 3.13.0rc2: | vs. base: |
|---|---|---|---|---|---|
| 2024-12-03 | python/bfb0788bfcaab7474c1b | bfb0788 | 1.059x ↑ 📄📈 |
1.021x ↑ 📄📈 |
|
| 2024-12-03 | python/bfb0788bfcaab7474c1b | bfb0788 (NOGIL) | 1.259x ↓ 📄📈 |
1.283x ↓ 📄📈 |
1.295x ↓ 📄📈🧠 |
| 2024-12-02 | nascheme/gh_115999_specialize | e517ccb (NOGIL) | 1.265x ↓ 📄📈 |
1.290x ↓ 📄📈 |
1.007x ↓ 📄📈🧠 |
| 2024-12-02 | nascheme/gh_115999_specialize | e517ccb | 1.030x ↑ 📄📈 |
1.007x ↓ 📄📈 |
1.001x ↓ 📄📈🧠 |
| 2024-12-01 | python/7ea523f47cdb4cf512a1 | 7ea523f (NOGIL) | 1.261x ↓ 📄📈 |
1.286x ↓ 📄📈 |
1.277x ↓ 📄📈🧠 |
| 2024-12-01 | python/7ea523f47cdb4cf512a1 | 7ea523f | 1.033x ↑ 📄📈 |
1.004x ↓ 📄📈 |
|
| 2024-12-01 | corona10/gh_115999_binary_sub | f0b8a8f | 1.036x ↑ 📄📈 |
1.002x ↓ 📄📈 |
1.001x ↑ 📄📈🧠 |
| 2024-11-30 | corona10/gh_115999_binary_sub | 0637d48 (NOGIL) | 1.254x ↓ 📄📈 |
1.280x ↓ 📄📈 |
1.012x ↑ 📄📈🧠 |
| 2024-11-30 | python/328187cc4fcdd578db42 | 328187c | 1.033x ↑ 📄📈 |
1.005x ↓ 📄📈 |
|
| 2024-11-30 | python/328187cc4fcdd578db42 | 328187c (NOGIL) | 1.254x ↓ 📄📈 |
1.279x ↓ 📄📈 |
1.270x ↓ 📄📈🧠 |
| 2024-11-29 | python/38264a060a8178d58046 | 38264a0 (NOGIL) | 1.261x ↓ 📄📈 |
1.286x ↓ 📄📈 |
1.276x ↓ 📄📈🧠 |
| 2024-11-29 | python/38264a060a8178d58046 | 38264a0 | 1.032x ↑ 📄📈 |
1.005x ↓ 📄📈 |
1.001x ↑ 📄📈🧠 |
| 2024-11-29 | python/15d6506d175780bb29e5 | 15d6506 | 1.031x ↑ 📄📈 |
1.007x ↓ 📄📈 |
|
| 2024-11-29 | python/15d6506d175780bb29e5 | 15d6506 (NOGIL) | 1.259x ↓ 📄📈 |
1.284x ↓ 📄📈 |
1.273x ↓ 📄📈🧠 |
| 2024-11-27 | nascheme/gh_127271_cell_get_s | b09c4cf | 1.036x ↑ 📄📈 |
1.002x ↓ 📄📈 |
1.004x ↓ 📄📈🧠 |
| 2024-11-26 | python/6da9d252ac39d5334245 | 6da9d25 | 1.040x ↑ 📄📈 |
1.002x ↑ 📄📈 |
* indicates that the exact same versions of pyperformance was not used.
Improvement of the geometric mean of key merged benchmarks, computed with pyperf compare.
The results have a resolution of 0.01 (1%).
Documentation
Running benchmarks from the GitHub web UI
Visit the 🔒 benchmark action and click the "Run Workflow" button.
The available parameters are:
fork: The fork of CPython to benchmark. If benchmarking a pull request, this would normally be your GitHub username.ref: The branch, tag or commit SHA to benchmark. If a SHA, it must be the full SHA, since finding it by a prefix is not supported.machine: The machine to run on. One oflinux-amd64(default),windows-amd64,darwin-arm64orall.benchmark_base: If checked, the base of the selected branch will also be benchmarked. The base is determined by runninggit merge-base upstream/main $ref.pystats: If checked, collect the pystats from running the benchmarks.
To watch the progress of the benchmark, select it from the 🔒 benchmark action page. It may be canceled from there as well. To show only your benchmark workflows, select your GitHub ID from the "Actor" dropdown.
When the benchmarking is complete, the results are published to this repository and will appear in the complete table. Each set of benchmarks will have:
- The raw
.jsonresults from pyperformance. - Comparisons against important reference releases, as well as the merge base of the branch if
benchmark_basewas selected. These include- A markdown table produced by
pyperf compare_to. - A set of "violin" plots showing the distribution of results for each benchmark.
- A markdown table produced by
The most convenient way to get results locally is to clone this repo and git pull from it.
Running benchmarks from the GitHub CLI
To automate benchmarking runs, it may be more convenient to use the GitHub CLI.
Once you have gh installed and configured, you can run benchmarks by cloning this repository and then from inside it:
gh workflow run benchmark.yml -f fork=me -f ref=my_branchAny of the parameters described above are available at the commandline using the -f key=value syntax.
Collecting Linux perf profiling data
To collect Linux perf sampling profile data for a benchmarking run, run the _benchmark action and check the perf checkbox.
Follow this by a run of the _generate action to regenerate the plots.
License
This repo is licensed under the BSD 3-Clause License, as found in the LICENSE file.