search

ionelmc / pytest-benchmark

py.test fixture for benchmarking code
BSD 2-Clause "Simplified" License
1.24k stars 119 forks source link

3.4.1: pytest is failing in few units #220

Closed kloczek closed 1 year ago

kloczek commented 2 years ago

I'm trying to package your module as an rpm package. So I'm using the typical PEP517 based build, install and test cycle used on building packages from non-root account.

Here is pytest output:

```console + PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-benchmark-3.4.1-6.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-pytest-benchmark-3.4.1-6.fc35.x86_64/usr/lib/python3.8/site-packages + /usr/bin/pytest -ra =========================================================================== test session starts ============================================================================ platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /home/tkloczko/rpmbuild/BUILD/pytest-benchmark-3.4.1, configfile: setup.cfg, testpaths: tests plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2 collected 232 items tests/test_benchmark.py ...............................................FFF.F. [ 22%] tests/test_calibration.py .................................. [ 37%] tests/test_cli.py ........... [ 42%] tests/test_doctest.rst . [ 42%] tests/test_elasticsearch_storage.py ....... [ 45%] tests/test_normal.py .......... [ 50%] tests/test_pedantic.py ................... [ 58%] tests/test_sample.py .... [ 59%] tests/test_skip.py s [ 60%] tests/test_stats.py .............. [ 66%] tests/test_storage.py ........................................ [ 83%] tests/test_utils.py .....ss..ss.s......s..s..s..s..... [ 98%] tests/test_with_testcase.py .. [ 99%] tests/test_with_weaver.py .. [100%] ================================================================================= FAILURES ================================================================================= ________________________________________________________________________________ test_basic ________________________________________________________________________________ /home/tkloczko/rpmbuild/BUILD/pytest-benchmark-3.4.1/tests/test_benchmark.py:1028: in test_basic result.stdout.fnmatch_lines([ E Failed: nomatch: '*collected 5 items' E and: '============================= test session starts ==============================' E and: 'platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3' E and: 'cachedir: .pytest_cache' E and: 'benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)' E and: 'rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_basic0' E and: 'plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2' E fnmatch: '*collected 5 items' E with: 'collecting ... collected 5 items' E nomatch: 'test_basic.py::*test_basic PASSED*' E and: '' E fnmatch: 'test_basic.py::*test_basic PASSED*' E with: 'test_basic.py::test_basic PASSED [ 20%]' E nomatch: 'test_basic.py::test_slow PASSED*' E and: 'test_basic.py::test_fast PASSED [ 40%]' E fnmatch: 'test_basic.py::test_slow PASSED*' E with: 'test_basic.py::test_slow PASSED [ 60%]' E fnmatch: 'test_basic.py::test_slower PASSED*' E with: 'test_basic.py::test_slower PASSED [ 80%]' E fnmatch: 'test_basic.py::test_xfast PASSED*' E with: 'test_basic.py::test_xfast PASSED [100%]' E nomatch: 'test_basic.py::test_fast PASSED*' E and: '' E and: '' E and: '--------------------------------------------------------------------------------------------------------- benchmark: 4 tests --------------------------------------------------------------------------------------------------------' E and: 'Name (time in ns) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations' E and: '-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------' E and: 'test_fast 60.3346 (1.0) 1,450.3254 (1.0) 67.0845 (1.0) 5.8999 (1.0) 66.8201 (1.0) 1.8242 (1.26) 1237;2742 14,906,574.9136 (1.0) 74289 200' E and: 'test_xfast 76.6944 (1.27) 17,735.1991 (12.23) 87.4878 (1.30) 49.7161 (8.43) 84.6456 (1.27) 1.4435 (1.0) 2830;23083 11,430,159.8760 (0.77) 161241 20' E and: 'test_slow 1,018,696.9303 (>1000.0) 1,077,774.9121 (743.13) 1,065,863.1824 (>1000.0) 7,461.3055 (>1000.0) 1,065,904.6238 (>1000.0) 5,922.5131 (>1000.0) 60;42 938.2067 (0.00) 944 1' E and: 'test_slower 10,143,317.0494 (>1000.0) 10,208,614.8690 (>1000.0) 10,148,093.7323 (>1000.0) 7,102.6432 (>1000.0) 10,146,973.1890 (>1000.0) 2,864.5154 (>1000.0) 3;3 98.5407 (0.00) 99 1' E and: '-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------' E and: '' E and: 'Legend:' E and: ' Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.' E and: ' OPS: Operations Per Second, computed as 1 / Mean' E and: '============================== 5 passed in 5.72s ===============================' E remains unmatched: 'test_basic.py::test_fast PASSED*' --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-102/test_basic0/runpytest-0 -vv --doctest-modules /tmp/pytest-of-tkloczko/pytest-102/test_basic0/test_basic.py in: /tmp/pytest-of-tkloczko/pytest-102/test_basic0 ============================= test session starts ============================== platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_basic0 plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2 collecting ... collected 5 items test_basic.py::test_basic PASSED [ 20%] test_basic.py::test_fast PASSED [ 40%] test_basic.py::test_slow PASSED [ 60%] test_basic.py::test_slower PASSED [ 80%] test_basic.py::test_xfast PASSED [100%] --------------------------------------------------------------------------------------------------------- benchmark: 4 tests -------------------------------------------------------------------------------------------------------- Name (time in ns) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- test_fast 60.3346 (1.0) 1,450.3254 (1.0) 67.0845 (1.0) 5.8999 (1.0) 66.8201 (1.0) 1.8242 (1.26) 1237;2742 14,906,574.9136 (1.0) 74289 200 test_xfast 76.6944 (1.27) 17,735.1991 (12.23) 87.4878 (1.30) 49.7161 (8.43) 84.6456 (1.27) 1.4435 (1.0) 2830;23083 11,430,159.8760 (0.77) 161241 20 test_slow 1,018,696.9303 (>1000.0) 1,077,774.9121 (743.13) 1,065,863.1824 (>1000.0) 7,461.3055 (>1000.0) 1,065,904.6238 (>1000.0) 5,922.5131 (>1000.0) 60;42 938.2067 (0.00) 944 1 test_slower 10,143,317.0494 (>1000.0) 10,208,614.8690 (>1000.0) 10,148,093.7323 (>1000.0) 7,102.6432 (>1000.0) 10,146,973.1890 (>1000.0) 2,864.5154 (>1000.0) 3;3 98.5407 (0.00) 99 1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean ============================== 5 passed in 5.72s =============================== ________________________________________________________________________________ test_skip _________________________________________________________________________________ /home/tkloczko/rpmbuild/BUILD/pytest-benchmark-3.4.1/tests/test_benchmark.py:1052: in test_skip result.stdout.fnmatch_lines([ E Failed: nomatch: '*collected 5 items' E and: '============================= test session starts ==============================' E and: 'platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3' E and: 'cachedir: .pytest_cache' E and: 'benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)' E and: 'rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_skip0' E and: 'plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2' E fnmatch: '*collected 5 items' E with: 'collecting ... collected 5 items' E nomatch: 'test_skip.py::*test_skip PASSED*' E and: '' E fnmatch: 'test_skip.py::*test_skip PASSED*' E with: 'test_skip.py::test_skip PASSED [ 20%]' E nomatch: 'test_skip.py::test_slow SKIPPED*' E and: 'test_skip.py::test_fast SKIPPED (Skipping benchmark (--benchmark-skip active).) [ 40%]' E fnmatch: 'test_skip.py::test_slow SKIPPED*' E with: 'test_skip.py::test_slow SKIPPED (Skipping benchmark (--benchmark-skip active).) [ 60%]' E fnmatch: 'test_skip.py::test_slower SKIPPED*' E with: 'test_skip.py::test_slower SKIPPED (Skipping benchmark (--benchmark-skip active).) [ 80%]' E fnmatch: 'test_skip.py::test_xfast SKIPPED*' E with: 'test_skip.py::test_xfast SKIPPED (Skipping benchmark (--benchmark-skip active).) [100%]' E nomatch: 'test_skip.py::test_fast SKIPPED*' E and: '' E and: '========================= 1 passed, 4 skipped in 0.07s =========================' E remains unmatched: 'test_skip.py::test_fast SKIPPED*' --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-102/test_skip0/runpytest-0 -vv --doctest-modules --benchmark-skip /tmp/pytest-of-tkloczko/pytest-102/test_skip0/test_skip.py in: /tmp/pytest-of-tkloczko/pytest-102/test_skip0 ============================= test session starts ============================== platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_skip0 plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2 collecting ... collected 5 items test_skip.py::test_skip PASSED [ 20%] test_skip.py::test_fast SKIPPED (Skipping benchmark (--benchmark-skip active).) [ 40%] test_skip.py::test_slow SKIPPED (Skipping benchmark (--benchmark-skip active).) [ 60%] test_skip.py::test_slower SKIPPED (Skipping benchmark (--benchmark-skip active).) [ 80%] test_skip.py::test_xfast SKIPPED (Skipping benchmark (--benchmark-skip active).) [100%] ========================= 1 passed, 4 skipped in 0.07s ========================= _______________________________________________________________________________ test_disable _______________________________________________________________________________ /home/tkloczko/rpmbuild/BUILD/pytest-benchmark-3.4.1/tests/test_benchmark.py:1066: in test_disable result.stdout.fnmatch_lines([ E Failed: nomatch: '*collected 5 items' E and: '============================= test session starts ==============================' E and: 'platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3' E and: 'cachedir: .pytest_cache' E and: 'benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)' E and: 'rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_disable0' E and: 'plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2' E fnmatch: '*collected 5 items' E with: 'collecting ... collected 5 items' E nomatch: 'test_disable.py::*test_disable PASSED*' E and: '' E fnmatch: 'test_disable.py::*test_disable PASSED*' E with: 'test_disable.py::test_disable PASSED [ 20%]' E nomatch: 'test_disable.py::test_slow PASSED*' E and: 'test_disable.py::test_fast PASSED [ 40%]' E fnmatch: 'test_disable.py::test_slow PASSED*' E with: 'test_disable.py::test_slow PASSED [ 60%]' E fnmatch: 'test_disable.py::test_slower PASSED*' E with: 'test_disable.py::test_slower PASSED [ 80%]' E fnmatch: 'test_disable.py::test_xfast PASSED*' E with: 'test_disable.py::test_xfast PASSED [100%]' E nomatch: 'test_disable.py::test_fast PASSED*' E and: '' E and: '============================== 5 passed in 0.08s ===============================' E remains unmatched: 'test_disable.py::test_fast PASSED*' --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-102/test_disable0/runpytest-0 -vv --doctest-modules --benchmark-disable /tmp/pytest-of-tkloczko/pytest-102/test_disable0/test_disable.py in: /tmp/pytest-of-tkloczko/pytest-102/test_disable0 ============================= test session starts ============================== platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_disable0 plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2 collecting ... collected 5 items test_disable.py::test_disable PASSED [ 20%] test_disable.py::test_fast PASSED [ 40%] test_disable.py::test_slow PASSED [ 60%] test_disable.py::test_slower PASSED [ 80%] test_disable.py::test_xfast PASSED [100%] ============================== 5 passed in 0.08s =============================== ___________________________________________________________________________ test_only_benchmarks ___________________________________________________________________________ /home/tkloczko/rpmbuild/BUILD/pytest-benchmark-3.4.1/tests/test_benchmark.py:1095: in test_only_benchmarks result.stdout.fnmatch_lines([ E Failed: nomatch: '*collected 5 items' E and: '============================= test session starts ==============================' E and: 'platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3' E and: 'cachedir: .pytest_cache' E and: 'benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)' E and: 'rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_only_benchmarks0' E and: 'plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2' E fnmatch: '*collected 5 items' E with: 'collecting ... collected 5 items' E nomatch: 'test_only_benchmarks.py::*test_only_benchmarks SKIPPED*' E and: '' E fnmatch: 'test_only_benchmarks.py::*test_only_benchmarks SKIPPED*' E with: 'test_only_benchmarks.py::test_only_benchmarks SKIPPED (Skipping non-benchmark (--benchmark-only active).) [ 20%]' E nomatch: 'test_only_benchmarks.py::test_slow PASSED*' E and: 'test_only_benchmarks.py::test_fast PASSED [ 40%]' E fnmatch: 'test_only_benchmarks.py::test_slow PASSED*' E with: 'test_only_benchmarks.py::test_slow PASSED [ 60%]' E fnmatch: 'test_only_benchmarks.py::test_slower PASSED*' E with: 'test_only_benchmarks.py::test_slower PASSED [ 80%]' E fnmatch: 'test_only_benchmarks.py::test_xfast PASSED*' E with: 'test_only_benchmarks.py::test_xfast PASSED [100%]' E nomatch: 'test_only_benchmarks.py::test_fast PASSED*' E and: '' E and: '' E and: '--------------------------------------------------------------------------------------------------------- benchmark: 4 tests --------------------------------------------------------------------------------------------------------' E and: 'Name (time in ns) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations' E and: '-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------' E and: 'test_fast 61.1693 (1.0) 2,699.3407 (1.0) 67.9581 (1.0) 8.3670 (1.0) 67.5698 (1.0) 1.5507 (1.24) 1109;4979 14,714,958.6013 (1.0) 149256 100' E and: 'test_xfast 81.6421 (1.33) 16,302.8482 (6.04) 87.7940 (1.29) 46.2650 (5.53) 84.6107 (1.25) 1.2456 (1.0) 2649;7987 11,390,302.1347 (0.77) 154802 20' E and: 'test_slow 1,018,308.1031 (>1000.0) 1,077,129.0399 (399.03) 1,065,356.0196 (>1000.0) 5,908.4934 (706.17) 1,064,853.0442 (>1000.0) 4,758.7091 (>1000.0) 56;42 938.6534 (0.00) 943 1' E and: 'test_slower 10,142,205.0502 (>1000.0) 10,177,019.9835 (>1000.0) 10,146,164.8010 (>1000.0) 5,441.6449 (650.37) 10,145,161.0681 (>1000.0) 2,474.4077 (>1000.0) 5;5 98.5594 (0.00) 99 1' E and: '-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------' E and: '' E and: 'Legend:' E and: ' Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.' E and: ' OPS: Operations Per Second, computed as 1 / Mean' E and: '========================= 4 passed, 1 skipped in 6.13s =========================' E remains unmatched: 'test_only_benchmarks.py::test_fast PASSED*' --------------------------------------------------------------------------- Captured stdout call --------------------------------------------------------------------------- running: /usr/bin/python3 -mpytest --basetemp=/tmp/pytest-of-tkloczko/pytest-102/test_only_benchmarks0/runpytest-0 -vv --doctest-modules --benchmark-only /tmp/pytest-of-tkloczko/pytest-102/test_only_benchmarks0/test_only_benchmarks.py in: /tmp/pytest-of-tkloczko/pytest-102/test_only_benchmarks0 ============================= test session starts ============================== platform linux -- Python 3.8.13, pytest-7.1.2, pluggy-1.0.0 -- /usr/bin/python3 cachedir: .pytest_cache benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000) rootdir: /tmp/pytest-of-tkloczko/pytest-102/test_only_benchmarks0 plugins: benchmark-3.4.1, forked-1.4.0, xdist-2.5.0, aspectlib-1.5.2 collecting ... collected 5 items test_only_benchmarks.py::test_only_benchmarks SKIPPED (Skipping non-benchmark (--benchmark-only active).) [ 20%] test_only_benchmarks.py::test_fast PASSED [ 40%] test_only_benchmarks.py::test_slow PASSED [ 60%] test_only_benchmarks.py::test_slower PASSED [ 80%] test_only_benchmarks.py::test_xfast PASSED [100%] --------------------------------------------------------------------------------------------------------- benchmark: 4 tests -------------------------------------------------------------------------------------------------------- Name (time in ns) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- test_fast 61.1693 (1.0) 2,699.3407 (1.0) 67.9581 (1.0) 8.3670 (1.0) 67.5698 (1.0) 1.5507 (1.24) 1109;4979 14,714,958.6013 (1.0) 149256 100 test_xfast 81.6421 (1.33) 16,302.8482 (6.04) 87.7940 (1.29) 46.2650 (5.53) 84.6107 (1.25) 1.2456 (1.0) 2649;7987 11,390,302.1347 (0.77) 154802 20 test_slow 1,018,308.1031 (>1000.0) 1,077,129.0399 (399.03) 1,065,356.0196 (>1000.0) 5,908.4934 (706.17) 1,064,853.0442 (>1000.0) 4,758.7091 (>1000.0) 56;42 938.6534 (0.00) 943 1 test_slower 10,142,205.0502 (>1000.0) 10,177,019.9835 (>1000.0) 10,146,164.8010 (>1000.0) 5,441.6449 (650.37) 10,145,161.0681 (>1000.0) 2,474.4077 (>1000.0) 5;5 98.5594 (0.00) 99 1 ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean ========================= 4 passed, 1 skipped in 6.13s ========================= ============================================================================= warnings summary ============================================================================= ../../../../../usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1198 /usr/lib/python3.8/site-packages/_pytest/config/__init__.py:1198: PytestRemovedIn8Warning: The --strict option is deprecated, use --strict-markers instead. self.issue_config_time_warning( tests/test_utils.py:35 /home/tkloczko/rpmbuild/BUILD/pytest-benchmark-3.4.1/tests/test_utils.py:35: PytestDeprecationWarning: @pytest.yield_fixture is deprecated. Use @pytest.fixture instead; they are the same. @pytest.yield_fixture(params=(True, False)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ----------------------------------------------------------------------------------------------------------------------------------- benchmark: 61 tests ------------------------------------------------------------------------------------------------------------------------------------ Name (time in ns) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- test_calibrate_stuck[False--1-1e-10] 5.0500 (1.0) 5.0500 (1.0) 5.0500 (1.0) 0.0000 (1.0) 5.0500 (1.0) 0.0000 (1.0) 0;0 198,019,801.9802 (1.0) 1 2 test_calibrate_stuck[False-0-1e-10] 5.0500 (1.0) 5.0500 (1.0) 5.0500 (1.0) 0.0000 (1.0) 5.0500 (1.0) 0.0000 (1.0) 0;0 198,019,801.9802 (1.0) 1 2 test_calibrate_stuck[False-1-1e-10] 5.0500 (1.0) 5.0500 (1.0) 5.0500 (1.0) 0.0000 (1.0) 5.0500 (1.0) 0.0000 (1.0) 0;0 198,019,801.9802 (1.0) 1 2 test_calibrate_stuck[True--1-1e-10] 10.0000 (1.98) 10.0000 (1.98) 10.0000 (1.98) 0.0000 (1.0) 10.0000 (1.98) 0.0000 (1.0) 0;0 100,000,000.0001 (0.51) 1 1 test_calibrate_stuck[True-0-1e-10] 10.0000 (1.98) 10.0000 (1.98) 10.0000 (1.98) 0.0000 (1.0) 10.0000 (1.98) 0.0000 (1.0) 0;0 100,000,000.0001 (0.51) 1 1 test_calibrate_stuck[True-1-1e-10] 10.0000 (1.98) 10.0000 (1.98) 10.0000 (1.98) 0.0000 (1.0) 10.0000 (1.98) 0.0000 (1.0) 0;0 100,000,000.0001 (0.51) 1 1 test_calibrate_stuck[False--1-1e-09] 50.5000 (10.00) 50.5000 (10.00) 50.5000 (10.00) 0.0000 (1.0) 50.5000 (10.00) 0.0000 (1.0) 0;0 19,801,980.1980 (0.10) 1 2 test_calibrate_stuck[False-0-1e-09] 50.5000 (10.00) 50.5000 (10.00) 50.5000 (10.00) 0.0000 (1.0) 50.5000 (10.00) 0.0000 (1.0) 0;0 19,801,980.1980 (0.10) 1 2 test_calibrate_stuck[False-1-1e-09] 50.5000 (10.00) 50.5000 (10.00) 50.5000 (10.00) 0.0000 (1.0) 50.5000 (10.00) 0.0000 (1.0) 0;0 19,801,980.1980 (0.10) 1 2 test_xfast 69.1414 (13.69) 236.0344 (46.74) 73.4206 (14.54) 7.6819 (inf) 71.5256 (14.16) 2.3842 (inf) 11;99 13,620,163.8363 (0.07) 1311 100 test_calibrate_stuck[True--1-1e-09] 100.0000 (19.80) 100.0000 (19.80) 100.0000 (19.80) 0.0000 (1.0) 100.0000 (19.80) 0.0000 (1.0) 0;0 10,000,000.0000 (0.05) 1 1 test_calibrate_stuck[True-0-1e-09] 100.0000 (19.80) 100.0000 (19.80) 100.0000 (19.80) 0.0000 (1.0) 100.0000 (19.80) 0.0000 (1.0) 0;0 10,000,000.0000 (0.05) 1 1 test_calibrate_stuck[True-1-1e-09] 100.0000 (19.80) 100.0000 (19.80) 100.0000 (19.80) 0.0000 (1.0) 100.0000 (19.80) 0.0000 (1.0) 0;0 10,000,000.0000 (0.05) 1 1 test_calibrate_xfast 105.3093 (20.85) 472.2713 (93.52) 109.2744 (21.64) 6.9606 (inf) 107.7703 (21.34) 0.9313 (inf) 26025;128937 9,151,274.9619 (0.05) 944095 100 test_warmup_rounds 190.9211 (37.81) 461.0047 (91.29) 262.7727 (52.03) 112.3474 (inf) 216.9982 (42.97) 95.2859 (inf) 1;1 3,805,570.8807 (0.02) 5 1 test_rounds_iterations 202.2833 (40.06) 664.0097 (131.49) 267.6699 (53.00) 118.0298 (inf) 226.0786 (44.77) 44.4881 (inf) 1;3 3,735,945.3414 (0.02) 15 10 test_rounds 270.0835 (53.48) 983.9423 (194.84) 370.6509 (73.40) 191.1361 (inf) 276.8356 (54.82) 152.2712 (inf) 1;1 2,697,956.7587 (0.01) 15 1 test_iterations 366.9178 (72.66) 366.9178 (72.66) 366.9178 (72.66) 0.0000 (1.0) 366.9178 (72.66) 0.0000 (1.0) 0;0 2,725,405.9877 (0.01) 1 10 test_proto[SimpleProxy] 392.9017 (77.80) 1,310.2079 (259.45) 416.9472 (82.56) 34.8125 (inf) 410.2010 (81.23) 6.8103 (inf) 4244;18896 2,398,385.3778 (0.01) 105999 20 test_proto[LocalsSimpleProxy] 402.4478 (79.69) 2,263.9986 (448.32) 422.6021 (83.68) 29.1095 (inf) 416.8483 (82.54) 2.6077 (inf) 3155;19782 2,366,292.0758 (0.01) 113790 20 test_calibrate_fast 446.4140 (88.40) 35,590.4146 (>1000.0) 469.0549 (92.88) 49.5077 (inf) 462.1688 (91.52) 8.4983 (inf) 13450;264784 2,131,946.5124 (0.01) 1861147 12 test_proto[CachedPropertyProxy] 477.0700 (94.47) 39,118.1093 (>1000.0) 534.8930 (105.92) 165.7659 (inf) 528.9912 (104.75) 28.8710 (inf) 120;1679 1,869,532.7258 (0.01) 135977 1 test_proto[LocalsCachedPropertyProxy] 484.0549 (95.85) 17,573.1257 (>1000.0) 544.4772 (107.82) 123.7358 (inf) 539.0029 (106.73) 30.9665 (inf) 113;1480 1,836,624.3445 (0.01) 95604 1 test_setup_many_rounds 491.0398 (97.24) 4,793.0516 (949.12) 1,011.9751 (200.39) 1,331.6067 (inf) 561.9368 (111.27) 173.9245 (inf) 1;1 988,166.5967 (0.00) 10 1 test_can_use_both_args_and_setup_without_return 1,020.9624 (202.17) 1,020.9624 (202.17) 1,020.9624 (202.17) 0.0000 (1.0) 1,020.9624 (202.17) 0.0000 (1.0) 0;0 979,468.0265 (0.00) 1 1 test_single 1,524.1094 (301.80) 1,524.1094 (301.80) 1,524.1094 (301.80) 0.0000 (1.0) 1,524.1094 (301.80) 0.0000 (1.0) 0;0 656,120.8824 (0.00) 1 1 test_args_kwargs 2,075.9180 (411.07) 2,075.9180 (411.07) 2,075.9180 (411.07) 0.0000 (1.0) 2,075.9180 (411.07) 0.0000 (1.0) 0;0 481,714.5913 (0.00) 1 1 test_setup_cprofile 2,321.0887 (459.62) 2,321.0887 (459.62) 2,321.0887 (459.62) 0.0000 (1.0) 2,321.0887 (459.62) 0.0000 (1.0) 0;0 430,832.3098 (0.00) 1 1 test_setup 6,689.2244 (>1000.0) 6,689.2244 (>1000.0) 6,689.2244 (>1000.0) 0.0000 (1.0) 6,689.2244 (>1000.0) 0.0000 (1.0) 0;0 149,494.1628 (0.00) 1 1 test_foo 11,507.8874 (>1000.0) 79,398.0435 (>1000.0) 62,879.4291 (>1000.0) 6,814.6337 (inf) 64,024.0032 (>1000.0) 120.1406 (inf) 628;1714 15,903.4523 (0.00) 13288 1 test_foo2 11,621.9744 (>1000.0) 78,012.0026 (>1000.0) 62,781.9145 (>1000.0) 7,251.1619 (inf) 64,044.0267 (>1000.0) 153.9011 (inf) 638;1475 15,928.1540 (0.00) 12943 1 test_fast 12,119.9992 (>1000.0) 155,504.0944 (>1000.0) 64,611.8516 (>1000.0) 6,944.7204 (inf) 65,039.1448 (>1000.0) 1,930.8645 (inf) 604;690 15,477.0367 (0.00) 13247 1 test_calibrate_slow 23,844.1862 (>1000.0) 288,401.9632 (>1000.0) 74,787.6053 (>1000.0) 3,565.6231 (inf) 75,085.0886 (>1000.0) 194.1808 (inf) 27250;105060 13,371.1996 (0.00) 420453 1 test_parametrized[0] 25,964.1092 (>1000.0) 145,361.0603 (>1000.0) 74,385.5100 (>1000.0) 4,563.7923 (inf) 74,869.9531 (>1000.0) 1,462.8749 (inf) 1129;1195 13,443.4784 (0.00) 14550 1 test_parametrized[1] 27,291.0111 (>1000.0) 165,853.1837 (>1000.0) 73,149.0805 (>1000.0) 6,085.6320 (inf) 73,449.9190 (>1000.0) 1,812.1209 (inf) 835;881 13,670.7118 (0.00) 11189 1 test_parametrized[4] 28,024.1948 (>1000.0) 149,097.0608 (>1000.0) 75,455.9719 (>1000.0) 2,043.1302 (inf) 74,934.9128 (>1000.0) 169.0350 (inf) 906;1346 13,252.7615 (0.00) 11206 1 test_parametrized[3] 29,809.0745 (>1000.0) 108,169.8574 (>1000.0) 75,487.9397 (>1000.0) 2,017.6242 (inf) 74,899.9882 (>1000.0) 189.0585 (inf) 925;1241 13,247.1492 (0.00) 11219 1 test_parametrized[2] 35,875.0112 (>1000.0) 114,708.9060 (>1000.0) 75,489.1851 (>1000.0) 2,167.4044 (inf) 74,883.9229 (>1000.0) 182.0736 (inf) 916;1265 13,246.9307 (0.00) 11162 1 test_calibrate 47,305.8317 (>1000.0) 2,130,588.0509 (>1000.0) 49,393.4140 (>1000.0) 18,655.2233 (inf) 48,498.1574 (>1000.0) 848.6677 (inf) 592;37414 20,245.6141 (0.00) 211256 1 test_slow 1,045,266.1663 (>1000.0) 1,181,653.0023 (>1000.0) 1,111,306.9231 (>1000.0) 33,580.4832 (inf) 1,135,065.9188 (>1000.0) 67,013.7815 (inf) 394;0 899.8414 (0.00) 930 1 test_weave_fixture 10,142,954.9977 (>1000.0) 10,150,033.0493 (>1000.0) 10,146,870.0450 (>1000.0) 2,692.2242 (inf) 10,146,907.0651 (>1000.0) 3,726.1634 (inf) 2;0 98.5526 (0.00) 5 1 test_slower 10,143,091.9021 (>1000.0) 10,168,803.8558 (>1000.0) 10,148,368.5007 (>1000.0) 3,077.4880 (inf) 10,148,223.9552 (>1000.0) 2,371.8458 (inf) 18;4 98.5380 (0.00) 99 1 test_weave_method 10,143,209.9473 (>1000.0) 10,150,093.1196 (>1000.0) 10,147,328.7679 (>1000.0) 2,631.0246 (inf) 10,147,650.9590 (>1000.0) 3,379.7114 (inf) 2;0 98.5481 (0.00) 5 1 test_calibrate_stuck[False--1-0.01] 504,999,999.9992 (>1000.0) 504,999,999.9992 (>1000.0) 504,999,999.9992 (>1000.0) 0.0000 (1.0) 504,999,999.9992 (>1000.0) 0.0000 (1.0) 0;0 1.9802 (0.00) 1 2 test_calibrate_stuck[False-0-0.01] 504,999,999.9992 (>1000.0) 504,999,999.9992 (>1000.0) 504,999,999.9992 (>1000.0) 0.0000 (1.0) 504,999,999.9992 (>1000.0) 0.0000 (1.0) 0;0 1.9802 (0.00) 1 2 test_calibrate_stuck[False-1-0.01] 504,999,999.9992 (>1000.0) 504,999,999.9992 (>1000.0) 504,999,999.9992 (>1000.0) 0.0000 (1.0) 504,999,999.9992 (>1000.0) 0.0000 (1.0) 0;0 1.9802 (0.00) 1 2 test_calibrate_stuck[True--1-0.01] 1,000,000,000.0000 (>1000.0) 1,000,000,000.0000 (>1000.0) 1,000,000,000.0000 (>1000.0) 0.0000 (1.0) 1,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 1.0000 (0.00) 1 1 test_calibrate_stuck[True-0-0.01] 1,000,000,000.0000 (>1000.0) 1,000,000,000.0000 (>1000.0) 1,000,000,000.0000 (>1000.0) 0.0000 (1.0) 1,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 1.0000 (0.00) 1 1 test_calibrate_stuck[True-1-0.01] 1,000,000,000.0000 (>1000.0) 1,000,000,000.0000 (>1000.0) 1,000,000,000.0000 (>1000.0) 0.0000 (1.0) 1,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 1.0000 (0.00) 1 1 test_calibrate_stuck[False--1-1.000000000000001] 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0198 (0.00) 1 2 test_calibrate_stuck[False--1-1] 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0198 (0.00) 1 2 test_calibrate_stuck[False-0-1.000000000000001] 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0198 (0.00) 1 2 test_calibrate_stuck[False-0-1] 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0198 (0.00) 1 2 test_calibrate_stuck[False-1-1.000000000000001] 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0198 (0.00) 1 2 test_calibrate_stuck[False-1-1] 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 50,500,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0198 (0.00) 1 2 test_calibrate_stuck[True--1-1.000000000000001] 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0100 (0.00) 1 1 test_calibrate_stuck[True--1-1] 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0100 (0.00) 1 1 test_calibrate_stuck[True-0-1.000000000000001] 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0100 (0.00) 1 1 test_calibrate_stuck[True-0-1] 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0100 (0.00) 1 1 test_calibrate_stuck[True-1-1.000000000000001] 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0100 (0.00) 1 1 test_calibrate_stuck[True-1-1] 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 100,000,000,000.0000 (>1000.0) 0.0000 (1.0) 0;0 0.0100 (0.00) 1 1 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Legend: Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile. OPS: Operations Per Second, computed as 1 / Mean --------------------------------------------------------------------------- cProfile (time in s) --------------------------------------------------------------------------- tests/test_pedantic.py::test_setup_cprofile ncalls tottime percall cumtime percall filename:lineno(function) 1 0.0000 0.0000 0.0000 0.0000 pytest-benchmark-3.4.1/tests/test_pedantic.py:29(stuff) 1 0.0000 0.0000 0.0000 0.0000 ~:0() 1 0.0000 0.0000 0.0000 0.0000 ~:0() ========================================================================= short test summary info ========================================================================== SKIPPED [1] tests/test_skip.py:5: bla SKIPPED [5] tests/test_utils.py:47: 'hg' not available on $PATH SKIPPED [4] tests/test_utils.py:160: 'hg' not available on $PATH FAILED tests/test_benchmark.py::test_basic - Failed: nomatch: '*collected 5 items' FAILED tests/test_benchmark.py::test_skip - Failed: nomatch: '*collected 5 items' FAILED tests/test_benchmark.py::test_disable - Failed: nomatch: '*collected 5 items' FAILED tests/test_benchmark.py::test_only_benchmarks - Failed: nomatch: '*collected 5 items' ==================================================== 4 failed, 218 passed, 10 skipped, 2 warnings in 431.23s (0:07:11) ===================================================== ```
kloczek commented 1 year ago

Closig to open new ticket against 4.0.0.