Open nedbat opened 2 years ago
If I mark a test as xfail (expected to fail), then flaky shouldn't re-run the test when it fails. Additionally, xfail tests are counted twice, while regular failing tests are counted once.
test_it.py:
import pytest @pytest.mark.xfail def test_xfail(): assert False def test_fail(): assert False
Run with pytest:
% pytest -vv --force-flaky ============================================ test session starts ============================================ platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/virtualenvs/tmp-d591713aeced424/bin/python cachedir: .pytest_cache rootdir: /private/tmp/flakybug plugins: flaky-3.7.0 collected 2 items test_it.py::test_xfail XFAIL [ 50%] test_it.py::test_xfail XFAIL [ 50%] test_it.py::test_fail FAILED [100%] ================================================= FAILURES ================================================== _________________________________________________ test_fail _________________________________________________ def test_fail(): > assert False E assert False test_it.py:8: AssertionError ===Flaky Test Report=== test_xfail failed (1 runs remaining out of 2). <class 'AssertionError'> assert False [<TracebackEntry /private/tmp/flakybug/test_it.py:5>] test_xfail failed; it passed 0 out of the required 1 times. <class 'AssertionError'> assert False [<TracebackEntry /private/tmp/flakybug/test_it.py:5>] test_fail failed (1 runs remaining out of 2). <class 'AssertionError'> assert False [<TracebackEntry /private/tmp/flakybug/test_it.py:8>] test_fail failed; it passed 0 out of the required 1 times. <class 'AssertionError'> assert False [<TracebackEntry /private/tmp/flakybug/test_it.py:8>] ===End Flaky Test Report=== ========================================== short test summary info ========================================== FAILED test_it.py::test_fail - assert False ======================================= 1 failed, 2 xfailed in 0.06s ========================================
The xfail test is listed twice in the -vv lines, and counted twice in the summary at the end.
% pip freeze attrs==21.2.0 flaky==3.7.0 iniconfig==1.1.1 packaging==21.3 pluggy==1.0.0 py==1.11.0 pyparsing==3.0.6 pytest==6.2.5 toml==0.10.2
If I mark a test as xfail (expected to fail), then flaky shouldn't re-run the test when it fails. Additionally, xfail tests are counted twice, while regular failing tests are counted once.
test_it.py:
Run with pytest:
The xfail test is listed twice in the -vv lines, and counted twice in the summary at the end.