box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
377 stars 60 forks source link

Flaky re-runs xfail tests #184

Open nedbat opened 2 years ago

nedbat commented 2 years ago

If I mark a test as xfail (expected to fail), then flaky shouldn't re-run the test when it fails. Additionally, xfail tests are counted twice, while regular failing tests are counted once.

test_it.py:

import pytest

@pytest.mark.xfail
def test_xfail():
    assert False

def test_fail():
    assert False

Run with pytest:

% pytest -vv --force-flaky
============================================ test session starts ============================================
platform darwin -- Python 3.9.7, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/virtualenvs/tmp-d591713aeced424/bin/python
cachedir: .pytest_cache
rootdir: /private/tmp/flakybug
plugins: flaky-3.7.0
collected 2 items

test_it.py::test_xfail XFAIL                                                                          [ 50%]
test_it.py::test_xfail XFAIL                                                                          [ 50%]
test_it.py::test_fail FAILED                                                                          [100%]

================================================= FAILURES ==================================================
_________________________________________________ test_fail _________________________________________________

    def test_fail():
>       assert False
E       assert False

test_it.py:8: AssertionError
===Flaky Test Report===

test_xfail failed (1 runs remaining out of 2).
    <class 'AssertionError'>
    assert False
    [<TracebackEntry /private/tmp/flakybug/test_it.py:5>]
test_xfail failed; it passed 0 out of the required 1 times.
    <class 'AssertionError'>
    assert False
    [<TracebackEntry /private/tmp/flakybug/test_it.py:5>]
test_fail failed (1 runs remaining out of 2).
    <class 'AssertionError'>
    assert False
    [<TracebackEntry /private/tmp/flakybug/test_it.py:8>]
test_fail failed; it passed 0 out of the required 1 times.
    <class 'AssertionError'>
    assert False
    [<TracebackEntry /private/tmp/flakybug/test_it.py:8>]

===End Flaky Test Report===
========================================== short test summary info ==========================================
FAILED test_it.py::test_fail - assert False
======================================= 1 failed, 2 xfailed in 0.06s ========================================

The xfail test is listed twice in the -vv lines, and counted twice in the summary at the end.

% pip freeze
attrs==21.2.0
flaky==3.7.0
iniconfig==1.1.1
packaging==21.3
pluggy==1.0.0
py==1.11.0
pyparsing==3.0.6
pytest==6.2.5
toml==0.10.2