pytest-dev / pytest-rerunfailures

a pytest plugin that re-runs failed tests up to -n times to eliminate flakey failures
Other
369 stars 82 forks source link

Multiple --rerun-except arguments are ANDed instead of ORed #225

Closed cfulljames closed 1 year ago

cfulljames commented 1 year ago

While working on #221, I noticed that when passing multiple --rerun-except arguments (or, now when passing a list to @pytest.mark.flaky(rerun_except=[...])), all failures are rerun unless they match all of the patterns.

With this test:

def test_fail():
    raise AssertionError()

Running with the example --rerun-except args from the readme gives:

$ pytest --reruns 5 --rerun-except AssertionError --rerun-except OSError
======================== test session starts =========================
platform linux -- Python 3.9.2, pytest-7.4.0, pluggy-1.2.0
rootdir: /tmp/test_rerun
plugins: rerunfailures-11.1.2
collected 1 item

test_rerun.py R                                                [100%]R [100%]R [100%]R [100%]R [100%]F [100%]

============================== FAILURES ==============================
_____________________________ test_fail ______________________________

    def test_fail():
>       raise AssertionError()
E       AssertionError

test_rerun.py:2: AssertionError
====================== short test summary info =======================
FAILED test_rerun.py::test_fail - AssertionError
===================== 1 failed, 5 rerun in 0.02s =====================

I think the intention in this case (as implied by the docs) is that the test should not be rerun, since the failure matches one of the --rerun-except arguments.

Currently the logic in _should_hard_fail_on_error is:

The fix for this isn't exactly obvious, as the intended relationship between --only-rerun and --rerun-except is a bit unclear when they are used together. My proposal would be:

icemac commented 1 year ago

Fix is released in https://pypi.org/project/pytest-rerunfailures/12.0/.