While working on #221, I noticed that when passing multiple --rerun-except arguments (or, now when passing a list to @pytest.mark.flaky(rerun_except=[...])), all failures are rerun unless they match all of the patterns.
With this test:
def test_fail():
raise AssertionError()
Running with the example --rerun-except args from the readme gives:
$ pytest --reruns 5 --rerun-except AssertionError --rerun-except OSError
======================== test session starts =========================
platform linux -- Python 3.9.2, pytest-7.4.0, pluggy-1.2.0
rootdir: /tmp/test_rerun
plugins: rerunfailures-11.1.2
collected 1 item
test_rerun.py R [100%]R [100%]R [100%]R [100%]R [100%]F [100%]
============================== FAILURES ==============================
_____________________________ test_fail ______________________________
def test_fail():
> raise AssertionError()
E AssertionError
test_rerun.py:2: AssertionError
====================== short test summary info =======================
FAILED test_rerun.py::test_fail - AssertionError
===================== 1 failed, 5 rerun in 0.02s =====================
I think the intention in this case (as implied by the docs) is that the test should not be rerun, since the failure matches one of the --rerun-except arguments.
If neither --only-rerun nor --rerun-except are used, all failures are rerun
Else, if one or more --only-rerun args is given, a failure is rerun if it matches any --only-rerun pattern
Else, if one or more --rerun-except args is given, a failure rerun if it fails to match any --rerun-except pattern
The fix for this isn't exactly obvious, as the intended relationship between --only-rerun and --rerun-except is a bit unclear when they are used together. My proposal would be:
If neither --only-rerun nor --rerun-except are used, all failures are rerun (no change)
Else, if --only-rerun is used alone, a failure is rerun if it matches any --only-rerun pattern (no change)
Else, if --rerun-except is used alone, a failure is rerun if it matches none of the --rerun-except patterns (new)
Else, if --only-rerun and --rerun-except are used together, a failure is only rerun if it matches at least one --only-rerun pattern, AND none of the --rerun-except patterns (new)
While working on #221, I noticed that when passing multiple --rerun-except arguments (or, now when passing a list to
@pytest.mark.flaky(rerun_except=[...])
), all failures are rerun unless they match all of the patterns.With this test:
Running with the example --rerun-except args from the readme gives:
I think the intention in this case (as implied by the docs) is that the test should not be rerun, since the failure matches one of the --rerun-except arguments.
Currently the logic in _should_hard_fail_on_error is:
The fix for this isn't exactly obvious, as the intended relationship between --only-rerun and --rerun-except is a bit unclear when they are used together. My proposal would be: