search

P403n1x87 / austin

Python frame stack sampler for CPython
https://pypi.org/project/austin-dist/
GNU General Public License v3.0
1.89k stars 54 forks source link

fix: version detection on Python 3.11 #184

Closed P403n1x87 closed 1 year ago

P403n1x87 commented 1 year ago

Description of the Change

The wrong symbol name was being used on OSX, preventing the inference of the Python version from symbols.

Regressions

None expected

Verification Process

Added a new test case to check that we get the correct full version from symbols.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage has no change and project coverage change: +0.72% :tada:

Comparison is base (456d787) 70.03% compared to head (891b3e1) 70.76%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## devel #184 +/- ## ========================================== + Coverage 70.03% 70.76% +0.72% ========================================== Files 25 25 Lines 2473 2473 Branches 730 730 ========================================== + Hits 1732 1750 +18 + Misses 414 392 -22 - Partials 327 331 +4 ``` | [Files Changed](https://app.codecov.io/gh/P403n1x87/austin/pull/184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Gabriele+N.+Tornetta) | Coverage Δ | | |---|---|---| | [src/python/symbols.h](https://app.codecov.io/gh/P403n1x87/austin/pull/184?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Gabriele+N.+Tornetta#diff-c3JjL3B5dGhvbi9zeW1ib2xzLmg=) | `100.00% <ø> (ø)` | | ... and [8 files with indirect coverage changes](https://app.codecov.io/gh/P403n1x87/austin/pull/184/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Gabriele+N.+Tornetta)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 1 year ago

Austin Benchmarks

Running Austin benchmarks with Python 3.10.12

Wall time [sampling interval: 1] ## Wall time [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 116000 ± 5000 | 1 ± 0 | 2e-05 ± 9e-06 | 13.5 ± 0.7 | | 3.5.0 | 116000 ± 5000 | 1 ± 0 | 2.1e-05 ± 7e-06 | 13.5 ± 0.7 | | dev | 116000 ± 5000 | 1 ± 0 | 1.9e-05 ± 6e-06 | 13.4 ± 0.7 |
Wall time [sampling interval: 10] ## Wall time [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 115000 ± 3000 | 0.545 ± 0.001 | 1.7e-05 ± 8e-06 | 13.5 ± 0.7 | | 3.5.0 | 115000 ± 4000 | 0.545 ± 0.002 | 1.8e-05 ± 8e-06 | 13.4 ± 0.7 | | dev | 112000 ± 5000 | 0.545 ± 0.002 | 1.6e-05 ± 8e-06 | 13.7 ± 0.8 |
Wall time [sampling interval: 100] ## Wall time [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 10700 ± 300 | 0.0007 ± 0.0002 | 0.00011 ± 4e-05 | 14.3 ± 0.5 | | 3.5.0 | 10700 ± 400 | 0.0008 ± 0.0002 | 7e-05 ± 6e-05 | 14.8 ± 0.6 | | dev | 10800 ± 300 | 0.0007 ± 0.0002 | 9e-05 ± 9e-05 | 14.2 ± 0.4 |
Wall time [sampling interval: 1000] ## Wall time [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 1850 ± 20 | 0 ± 0 | 0.0 ± 0.0001 | 20 ± 1 | | 3.5.0 | 1850 ± 20 | 0 ± 0 | 0.0001 ± 0.0002 | 18.3 ± 0.5 | | dev | 1840 ± 20 | 0.0002 ± 0.0003 | 0.0002 ± 0.0002 | 18.8 ± 0.9 |
CPU time [sampling interval: 1] ## CPU time [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 50000 ± 6000 | 1 ± 0 | 4e-05 ± 2e-05 | 23 ± 1 | | 3.5.0 | 51000 ± 3000 | 1 ± 0 | 1.9e-05 ± 9e-06 | 23 ± 1 | | dev | 54000 ± 2000 | 1 ± 0 | 2e-05 ± 2e-05 | 22.2 ± 0.4 |
CPU time [sampling interval: 10] ## CPU time [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 53000 ± 2000 | 0.9967 ± 0.0004 | 2e-05 ± 2e-05 | 22.5 ± 0.7 | | 3.5.0 | 52000 ± 3000 | 0.9966 ± 0.0004 | 2e-05 ± 1e-05 | 22.5 ± 0.7 | | dev | 53000 ± 3000 | 0.9964 ± 0.0004 | 3e-05 ± 2e-05 | 22.4 ± 1.0 |
CPU time [sampling interval: 100] ## CPU time [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 5430 ± 100 | 0.0013 ± 0.0004 | 0.0002 ± 0.0001 | 22.7 ± 0.5 | | 3.5.0 | 5500 ± 300 | 0.001 ± 0.0004 | 9e-05 ± 9e-05 | 22.5 ± 0.5 | | dev | 5500 ± 100 | 0.0011 ± 0.0003 | 0.0002 ± 0.0002 | 22.2 ± 0.4 |
CPU time [sampling interval: 1000] ## CPU time [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 951 ± 5 | 0.0001 ± 0.0002 | 0.0001 ± 0.0003 | 27.9 ± 1.0 | | 3.5.0 | 951 ± 4 | 0 ± 0 | 0 ± 0 | 27.5 ± 0.8 | | dev | 950 ± 10 | 0.0001 ± 0.0005 | 0.0001 ± 0.0002 | 29 ± 2 |
RSA keygen [sampling interval: 1] ## RSA keygen [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 40000 ± 1000 | 1 ± 0 | 0.0002 ± 0.0003 | 24.6 ± 0.8 | | 3.5.0 | 39700 ± 900 | 1 ± 0 | 0.0002 ± 0.0002 | 24.7 ± 0.7 | | dev | 39000 ± 2000 | 1 ± 0 | 0.0004 ± 0.0004 | 24.4 ± 0.7 |
RSA keygen [sampling interval: 10] ## RSA keygen [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 39400 ± 800 | 0.983 ± 0.01 | 0.0004 ± 0.0003 | 24.8 ± 0.6 | | 3.5.0 | 40000 ± 1000 | 0.988 ± 0.007 | 0.0002 ± 0.0001 | 24.5 ± 0.8 | | dev | 40000 ± 1000 | 0.98 ± 0.02 | 0.0003 ± 0.0003 | 24.4 ± 0.7 |
RSA keygen [sampling interval: 100] ## RSA keygen [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 6270 ± 40 | 0.002 ± 0.001 | 0.0006 ± 0.0004 | 28 ± 2 | | 3.5.0 | 6260 ± 50 | 0.002 ± 0.001 | 0.0008 ± 0.0005 | 29 ± 2 | | dev | 6270 ± 40 | 0.003 ± 0.002 | 0.001 ± 0.0009 | 27 ± 3 |
RSA keygen [sampling interval: 1000] ## RSA keygen [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 942 ± 3 | 0 ± 0 | 0.001 ± 0.003 | 32 ± 3 | | 3.5.0 | 943.1 ± 0.6 | 0 ± 0 | 0.001 ± 0.002 | 33 ± 2 | | dev | 942.3 ± 0.8 | 0 ± 0 | 0.001 ± 0.002 | 32 ± 3 |
Full metrics [sampling interval: 1] ## Full metrics [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 51000 ± 2000 | 1 ± 0 | 4e-05 ± 2e-05 | 30 ± 1 | | 3.5.0 | 51000 ± 1000 | 1 ± 0 | 4e-05 ± 2e-05 | 29.8 ± 0.9 | | dev | 50000 ± 2000 | 1 ± 0 | 5e-05 ± 2e-05 | 30 ± 2 |
Full metrics [sampling interval: 10] ## Full metrics [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 49000 ± 3000 | 1 ± 0 | 5e-05 ± 2e-05 | 31 ± 2 | | 3.5.0 | 51000 ± 3000 | 1 ± 0 | 4e-05 ± 2e-05 | 30 ± 3 | | dev | 52000 ± 1000 | 1 ± 0 | 4e-05 ± 2e-05 | 29.4 ± 0.7 |
Full metrics [sampling interval: 100] ## Full metrics [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 9900 ± 300 | 0.002 ± 0.0003 | 9e-05 ± 9e-05 | 34 ± 1 | | 3.5.0 | 10000 ± 500 | 0.004 ± 0.006 | 0.0001 ± 9e-05 | 34 ± 2 | | dev | 10000 ± 300 | 0.002 ± 0.0005 | 8e-05 ± 8e-05 | 33.1 ± 1.0 |
Full metrics [sampling interval: 1000] ## Full metrics [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 1850 ± 20 | 0.0001 ± 0.0002 | 0.0001 ± 0.0002 | 38 ± 2 | | 3.5.0 | 1860 ± 20 | 0.0001 ± 0.0003 | 0.0001 ± 0.0002 | 38.2 ± 0.9 | | dev | 1860 ± 20 | 0 ± 0 | 0.0001 ± 0.0002 | 38.8 ± 0.6 |
Multiprocess wall time [sampling interval: 1] ## Multiprocess wall time [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 2800 ± 200 | 1 ± 0 | 0.00014 ± 6e-05 | 330 ± 20 | | 3.5.0 | 3300 ± 200 | 1 ± 0 | 0.00015 ± 4e-05 | 290 ± 20 | | dev | 3300 ± 200 | 1 ± 0 | 0.00015 ± 4e-05 | 290 ± 20 |
Multiprocess wall time [sampling interval: 10] ## Multiprocess wall time [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 2800 ± 100 | 1 ± 0 | 0.00017 ± 3e-05 | 340 ± 10 | | 3.5.0 | 3200 ± 100 | 1 ± 0 | 0.00015 ± 4e-05 | 300 ± 10 | | dev | 3200 ± 200 | 1 ± 0 | 0.00017 ± 6e-05 | 300 ± 20 |
Multiprocess wall time [sampling interval: 100] ## Multiprocess wall time [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 2600 ± 100 | 0.04 ± 0.02 | 7e-05 ± 3e-05 | 360 ± 20 | | 3.5.0 | 3200 ± 200 | 0.04 ± 0.02 | 9e-05 ± 4e-05 | 300 ± 20 | | dev | 3100 ± 100 | 0.056 ± 0.004 | 8e-05 ± 3e-05 | 310 ± 10 |
Multiprocess wall time [sampling interval: 1000] ## Multiprocess wall time [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 2600 ± 200 | 0.015 ± 0.003 | 2e-05 ± 2e-05 | 120 ± 10 | | 3.5.0 | 3000 ± 100 | 0.008 ± 0.002 | 2e-05 ± 2e-05 | 72 ± 7 | | dev | 2950 ± 100 | 0.007 ± 0.001 | 2e-05 ± 2e-05 | 68 ± 6 |

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

Sample Rate Saturation Error Rate Sampling Speed
Wall time [sampling interval: 100] :yellow_circle: :yellow_circle: :yellow_circle: :green_circle:
CPU time [sampling interval: 1] :green_circle: :yellow_circle: :yellow_circle: :yellow_circle:
RSA keygen [sampling interval: 1000] :red_circle: :yellow_circle: :yellow_circle: :yellow_circle:
Multiprocess wall time [sampling interval: 100] :yellow_circle: :red_circle: :yellow_circle: :yellow_circle: