search

ionelmc / pytest-benchmark

py.test fixture for benchmarking code
BSD 2-Clause "Simplified" License
1.22k stars 115 forks source link

[self-tests] tests fail against Pytest 7 #214

Open stanislavlevin opened 2 years ago

stanislavlevin commented 2 years ago

With Pytest 7 benchmark tests fail:

FAILED tests/test_benchmark.py::test_basic - Failed: nomatch: '*collected 5 i...
FAILED tests/test_benchmark.py::test_skip - Failed: nomatch: '*collected 5 it...
FAILED tests/test_benchmark.py::test_disable - Failed: nomatch: '*collected 5...
FAILED tests/test_benchmark.py::test_only_benchmarks - Failed: nomatch: '*col...
====== 4 failed, 216 passed, 12 skipped, 5 warnings in 389.22s (0:06:29) =======

failure report for one test:

=================================== FAILURES ===================================
__________________________________ test_basic __________________________________
/usr/src/RPM/BUILD/python3-module-pytest-benchmark-3.4.1/tests/test_benchmark.py:1028:
 in test_basic
    result.stdout.fnmatch_lines([
E   Failed: nomatch: '*collected 5 items'
E       and: '============================= test session starts ======================
========'
E       and: 'platform linux -- Python 3.10.2, pytest-7.0.1, pluggy-1.0.0 -- /usr/src/
RPM/BUILD/python3-module-pytest-benchmark-3.4.1/.tox/py310-nocov/bin/python'
E       and: 'cachedir: .pytest_cache'
E       and: 'benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min
_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_
iterations=100000)'
E       and: 'rootdir: /usr/src/tmp/pytest-of-builder/pytest-0/test_basic0'
E       and: 'plugins: benchmark-3.4.1, aspectlib-1.4.2'
E   fnmatch: '*collected 5 items'
E      with: 'collecting ... collected 5 items'
E   nomatch: 'test_basic.py::*test_basic PASSED*'
E       and: ''
E   fnmatch: 'test_basic.py::*test_basic PASSED*'
E      with: 'test_basic.py::test_basic PASSED                                        
 [ 20%]'
E   nomatch: 'test_basic.py::test_slow PASSED*'
E       and: 'test_basic.py::test_fast PASSED                                         
 [ 40%]'
E   fnmatch: 'test_basic.py::test_slow PASSED*'
E      with: 'test_basic.py::test_slow PASSED                                         
 [ 60%]'
E   fnmatch: 'test_basic.py::test_slower PASSED*'
E      with: 'test_basic.py::test_slower PASSED                                       
 [ 80%]'
E   fnmatch: 'test_basic.py::test_xfast PASSED*'
E      with: 'test_basic.py::test_xfast PASSED                                        
 [100%]'
E   nomatch: 'test_basic.py::test_fast PASSED*'
E       and: ''
E       and: ''
E       and: '------------------------------------------------------------------------

...

E   remains unmatched: 'test_basic.py::test_fast PASSED*'

...

----------------------------- Captured stdout call -----------------------------
running: /usr/src/RPM/BUILD/python3-module-pytest-benchmark-3.4.1/.tox/py310-nocov/bin/python -mpytest --basetemp=/usr/src/tmp/pytest-of-builder/pytest-0/test_basic0/runpytest-0 -vv --doctest-modules /usr/src/tmp/pytest-of-builder/pytest-0/test_basic0/test_basic.py
     in: /usr/src/tmp/pytest-of-builder/pytest-0/test_basic0
============================= test session starts ==============================
platform linux -- Python 3.10.2, pytest-7.0.1, pluggy-1.0.0 -- /usr/src/RPM/BUILD/python3-module-pytest-benchmark-3.4.1/.tox/py310-nocov/bin/python
cachedir: .pytest_cache
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /usr/src/tmp/pytest-of-builder/pytest-0/test_basic0
plugins: benchmark-3.4.1, aspectlib-1.4.2
collecting ... collected 5 items

test_basic.py::test_basic PASSED                                         [ 20%]
test_basic.py::test_fast PASSED                                          [ 40%]
test_basic.py::test_slow PASSED                                          [ 60%]
test_basic.py::test_slower PASSED                                        [ 80%]
test_basic.py::test_xfast PASSED                                         [100%]
...

The actual order of reported passed tests differs from the expected one. This is due to pytest-dev/pytest#5196: Tests are now ordered by definition order in more cases.

I think there is no point to check the behavior of duplicated tests functions and I can only assume that originally this was a typo. In my opinion - there are 2 solutions - removal of the first duplicated test_fast function or renaming it and syncing all affected tests' assumptions.

See also https://github.com/pytest-dev/pytest/issues/9724

CyberTailor commented 2 years ago

I have more test failures for that reason:

FAILED tests/test_benchmark.py::test_groups - Failed: nomatch: '*collected 5 items'
FAILED tests/test_benchmark.py::test_group_by_func - Failed: fnmatch: '*'
FAILED tests/test_benchmark.py::test_group_by_fullfunc - Failed: fnmatch: '*'
FAILED tests/test_benchmark.py::test_group_by_param_all - Failed: fnmatch: '*'
FAILED tests/test_benchmark.py::test_group_by_param_select - Failed: fnmatch: '*'
FAILED tests/test_benchmark.py::test_group_by_param_select_multiple - Failed: fnmatch: '*'
FAILED tests/test_benchmark.py::test_group_by_fullname - Failed: matched:  "* benchmark 'test_x.py::test_a[[]0[]]': 1 tests *"
FAILED tests/test_benchmark.py::test_only_override_skip - Failed: nomatch: '*collected 2 items'
FAILED tests/test_benchmark.py::test_fixtures_also_skipped - Failed: nomatch: '*collected 2 items'
FAILED tests/test_benchmark.py::test_max_time_min_rounds - Failed: nomatch: '*collected 3 items'
FAILED tests/test_benchmark.py::test_max_time - Failed: nomatch: '*collected 3 items'
FAILED tests/test_benchmark.py::test_disable_gc - Failed: nomatch: '*collected 2 items'
FAILED tests/test_benchmark.py::test_custom_timer - Failed: nomatch: '*collected 2 items'
FAILED tests/test_benchmark.py::test_sort_by_mean - Failed: nomatch: '*collected 2 items'
FAILED tests/test_benchmark.py::test_basic - Failed: nomatch: '*collected 5 items'
FAILED tests/test_benchmark.py::test_skip - Failed: nomatch: '*collected 5 items'
FAILED tests/test_benchmark.py::test_disable - Failed: nomatch: '*collected 5 items'
FAILED tests/test_benchmark.py::test_mark_selection - Failed: nomatch: '*collected 5 items*'
FAILED tests/test_benchmark.py::test_only_benchmarks - Failed: nomatch: '*collected 5 items'