search

P403n1x87 / austin

Python frame stack sampler for CPython
https://pypi.org/project/austin-dist/
GNU General Public License v3.0
1.71k stars 51 forks source link

tests: run uwsgi tests in virtual environment #191

Closed P403n1x87 closed 1 year ago

P403n1x87 commented 1 year ago

Description of the Change

We run the uwsgi tests in separated virtual environment to ensure that we install it using the interpreter that Austin is being tested against. Currently, we install the dependency in the environment used to run pytest, which means that we test uwsgi with the same Python version in every job.

Regressions

We make the uwsgi tests run as expected, so there is always the chance of new CI findings.

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage has no change and project coverage change: -0.77% :warning:

Comparison is base (55f56fb) 68.04% compared to head (614ed45) 67.28%.

:exclamation: Current head 614ed45 differs from pull request most recent head 7a628a9. Consider uploading reports for the commit 7a628a9 to get more accurate results

Additional details and impacted files ```diff @@ Coverage Diff @@ ## devel #191 +/- ## ========================================== - Coverage 68.04% 67.28% -0.77% ========================================== Files 27 27 Lines 2485 2485 Branches 761 761 ========================================== - Hits 1691 1672 -19 - Misses 458 471 +13 - Partials 336 342 +6 ``` [see 3 files with indirect coverage changes](https://app.codecov.io/gh/P403n1x87/austin/pull/191/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Gabriele+N.+Tornetta)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

github-actions[bot] commented 1 year ago

Austin Benchmarks

Running Austin benchmarks with Python 3.10.12

Benchmark Summary

Comparison of dev against 3.5.0.

The following scenarios show a statistically significant difference in performance between the two versions.

Sample Rate Saturation Error Rate Sampling Speed
RSA keygen [sampling interval: 100] :yellow_circle: :yellow_circle: :yellow_circle: :red_circle:
Full metrics [sampling interval: 100] :yellow_circle: :yellow_circle: :green_circle: :yellow_circle:
Full metrics [sampling interval: 1000] :yellow_circle: :yellow_circle: :yellow_circle: :green_circle:
Multiprocess wall time [sampling interval: 100] :yellow_circle: :yellow_circle: :yellow_circle: :green_circle:

Benchmark Results

Wall time [sampling interval: 1] ## Wall time [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 118000 ± 3000 | 1 ± 0 | 1.3e-05 ± 9e-06 | 13.1 ± 0.3 | | 3.5.0 | 121000 ± 1000 | 1 ± 0 | 1e-05 ± 4e-06 | 13 ± 0 | | dev | 120000 ± 3000 | 1 ± 0 | 9e-06 ± 4e-06 | 13.1 ± 0.3 |
Wall time [sampling interval: 10] ## Wall time [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 115000 ± 4000 | 0.543 ± 0.002 | 1.3e-05 ± 4e-06 | 13.3 ± 0.5 | | 3.5.0 | 114000 ± 5000 | 0.545 ± 0.001 | 9e-06 ± 4e-06 | 13.8 ± 0.6 | | dev | 116000 ± 6000 | 0.545 ± 0.0007 | 1.1e-05 ± 7e-06 | 13.4 ± 0.5 |
Wall time [sampling interval: 100] ## Wall time [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 10200 ± 200 | 0.0011 ± 0.0009 | 3e-05 ± 3e-05 | 14.7 ± 0.5 | | 3.5.0 | 10000 ± 400 | 0.0007 ± 0.0002 | 3e-05 ± 3e-05 | 14.4 ± 0.7 | | dev | 10300 ± 300 | 0.0008 ± 0.0002 | 4e-05 ± 3e-05 | 14.1 ± 0.3 |
Wall time [sampling interval: 1000] ## Wall time [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 1860 ± 20 | 0.0001 ± 0.0002 | 2e-05 ± 7e-05 | 18 ± 1 | | 3.5.0 | 1850 ± 20 | 0.0 ± 0.0001 | 0.0001 ± 0.0001 | 17.1 ± 0.3 | | dev | 1860 ± 20 | 0 ± 0 | 0.0001 ± 0.0001 | 16.9 ± 0.3 |
CPU time [sampling interval: 1] ## CPU time [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 51000 ± 4000 | 1 ± 0 | 1.9e-05 ± 8e-06 | 23 ± 1 | | 3.5.0 | 52000 ± 3000 | 1 ± 0 | 1.8e-05 ± 7e-06 | 22.5 ± 1.0 | | dev | 54000 ± 2000 | 1 ± 0 | 1.5e-05 ± 7e-06 | 22.0 ± 0.7 |
CPU time [sampling interval: 10] ## CPU time [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 54000 ± 2000 | 0.99795 ± 7e-05 | 1.3e-05 ± 6e-06 | 22.2 ± 0.4 | | 3.5.0 | 53000 ± 2000 | 0.998 ± 0.0002 | 1.5e-05 ± 1e-05 | 22.3 ± 0.5 | | dev | 53000 ± 2000 | 0.9979 ± 0.0006 | 1.1e-05 ± 1e-05 | 22.3 ± 0.8 |
CPU time [sampling interval: 100] ## CPU time [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 5350 ± 90 | 0.0011 ± 0.0002 | 9e-05 ± 6e-05 | 22.8 ± 0.8 | | 3.5.0 | 5400 ± 100 | 0.0012 ± 0.0003 | 0.0001 ± 0.0001 | 22.4 ± 0.5 | | dev | 5400 ± 100 | 0.0012 ± 0.0003 | 0.00011 ± 9e-05 | 22.5 ± 0.7 |
CPU time [sampling interval: 1000] ## CPU time [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 950 ± 1 | 0.0 ± 0.0001 | 0.0 ± 0.0001 | 26.1 ± 0.3 | | 3.5.0 | 951 ± 2 | 0.0001 ± 0.0003 | 0.0 ± 0.0001 | 26.1 ± 0.9 | | dev | 951 ± 2 | 0 ± 0 | 0.0002 ± 0.0002 | 25.9 ± 0.3 |
RSA keygen [sampling interval: 1] ## RSA keygen [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 39000 ± 1000 | 1 ± 0 | 0.0005 ± 0.0006 | 25 ± 1 | | 3.5.0 | 39700 ± 700 | 1 ± 0 | 0.0003 ± 0.0002 | 24.4 ± 0.7 | | dev | 39000 ± 1000 | 1 ± 0 | 0.0002 ± 0.0002 | 24.6 ± 0.8 |
RSA keygen [sampling interval: 10] ## RSA keygen [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 39500 ± 400 | 0.98 ± 0.01 | 0.0002 ± 0.0002 | 24.9 ± 0.3 | | 3.5.0 | 40000 ± 1000 | 0.98 ± 0.02 | 0.0002 ± 0.0001 | 24.1 ± 0.9 | | dev | 40000 ± 1000 | 0.98 ± 0.01 | 0.0002 ± 0.0001 | 23.8 ± 0.6 |
RSA keygen [sampling interval: 100] ## RSA keygen [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 6230 ± 70 | 0.003 ± 0.003 | 0.0008 ± 0.0007 | 30 ± 2 | | 3.5.0 | 6250 ± 30 | 0.0018 ± 0.0006 | 0.0004 ± 0.0003 | 29 ± 2 | | dev | 6240 ± 40 | 0.002 ± 0.002 | 0.0004 ± 0.0008 | 31 ± 2 |
RSA keygen [sampling interval: 1000] ## RSA keygen [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 943 ± 1 | 0 ± 0 | 0.001 ± 0.002 | 32 ± 2 | | 3.5.0 | 943.2 ± 0.9 | 0 ± 0 | 0.001 ± 0.002 | 32 ± 3 | | dev | 940 ± 5 | 0 ± 0 | 0.0 ± 0.001 | 31 ± 1 |
Full metrics [sampling interval: 1] ## Full metrics [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 50000 ± 3000 | 1 ± 0 | 3e-05 ± 1e-05 | 31 ± 2 | | 3.5.0 | 51900 ± 800 | 1 ± 0 | 3e-05 ± 1e-05 | 29.6 ± 0.5 | | dev | 52000 ± 1000 | 1 ± 0 | 3e-05 ± 1e-05 | 29.6 ± 0.8 |
Full metrics [sampling interval: 10] ## Full metrics [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 52000 ± 2000 | 1 ± 0 | 3.2e-05 ± 1e-05 | 30 ± 2 | | 3.5.0 | 52000 ± 2000 | 1 ± 0 | 2e-05 ± 1e-05 | 30 ± 2 | | dev | 52000 ± 1000 | 1 ± 0 | 2e-05 ± 1e-05 | 29.3 ± 0.9 |
Full metrics [sampling interval: 100] ## Full metrics [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 10000 ± 300 | 0.0018 ± 0.0004 | 8e-05 ± 7e-05 | 33 ± 1 | | 3.5.0 | 10000 ± 200 | 0.0018 ± 0.0004 | 8e-05 ± 5e-05 | 33 ± 1 | | dev | 9900 ± 200 | 0.002 ± 0.001 | 1e-05 ± 2e-05 | 32.6 ± 0.7 |
Full metrics [sampling interval: 1000] ## Full metrics [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 1870 ± 20 | 0.0003 ± 0.0003 | 0.0001 ± 0.0001 | 37.3 ± 0.8 | | 3.5.0 | 1860 ± 20 | 0.0002 ± 0.0003 | 0.0001 ± 0.0002 | 37.3 ± 0.7 | | dev | 1870 ± 10 | 0.0002 ± 0.0002 | 0.0001 ± 0.0002 | 36.5 ± 0.7 |
Multiprocess wall time [sampling interval: 1] ## Multiprocess wall time [sampling interval: 1] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 3100 ± 200 | 1 ± 0 | 0.00017 ± 4e-05 | 300 ± 20 | | 3.5.0 | 3400 ± 200 | 1 ± 0 | 0.00016 ± 3e-05 | 280 ± 10 | | dev | 3400 ± 200 | 1 ± 0 | 0.00014 ± 4e-05 | 280 ± 10 |
Multiprocess wall time [sampling interval: 10] ## Multiprocess wall time [sampling interval: 10] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 3000 ± 200 | 1 ± 0 | 0.00016 ± 3e-05 | 310 ± 20 | | 3.5.0 | 3300 ± 100 | 1 ± 0 | 0.00015 ± 4e-05 | 290 ± 10 | | dev | 3320 ± 100 | 1 ± 0 | 0.00015 ± 4e-05 | 288 ± 9 |
Multiprocess wall time [sampling interval: 100] ## Multiprocess wall time [sampling interval: 100] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 2800 ± 200 | 0.05 ± 0.02 | 9e-05 ± 4e-05 | 330 ± 30 | | 3.5.0 | 3300 ± 100 | 0.05 ± 0.01 | 0.0001 ± 0.0002 | 290 ± 10 | | dev | 3400 ± 100 | 0.05 ± 0.01 | 5e-05 ± 3e-05 | 278 ± 8 |
Multiprocess wall time [sampling interval: 1000] ## Multiprocess wall time [sampling interval: 1000] | | Sample Rate | Saturation | Error Rate | Sampling Speed | | --- |:-----------:|:----------:|:----------:|:--------------:| | 3.4.1 | 2700 ± 100 | 0.014 ± 0.004 | 1e-05 ± 2e-05 | 110 ± 10 | | 3.5.0 | 3100 ± 100 | 0.007 ± 0.002 | 1e-05 ± 2e-05 | 65 ± 7 | | dev | 3100 ± 200 | 0.007 ± 0.001 | 1e-05 ± 1e-05 | 64 ± 5 |