Don't rerun xfail tests

[hugovk]

I have tests which I know will fail and have marked them to xfail with pytest, because I'd still like them to run.

However, I know they will fail, so don't need flaky to re-run these.

Here's a minimal example:

import pytest
from flaky import flaky

@flaky(max_runs=3, min_passes=1)
class TestStuff:
    def test_ok(self):
        # Expect to run once and pass (it does)
        assert True

    def test_fail(self):
        # Expect to run three times and fail (it does)
        assert False

    @pytest.mark.xfail
    def test_xfail(self):
        # Expect to run once and fail (it runs three times and fails)
        assert False

pytest -v test_flaky_xfail.py
============================================================ test session starts ============================================================
platform darwin -- Python 3.8.3, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
cachedir: .pytest_cache
Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket=<bucket_type>
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/hugo/github/pylast/.hypothesis/examples')
rootdir: /Users/hugo/github/pylast, inifile: pytest.ini
plugins: xdist-2.0.0, reverse-1.0.1, requests-mock-1.8.0, random-order-1.0.4, flaky-3.7.0, hypothesis-5.23.3, cov-2.10.0, testmon-1.0.2, mypy-0.4.2, forked-1.1.3
collected 3 items

test_flaky_xfail.py::TestStuff::test_ok PASSED                                                                                        [ 33%]
test_flaky_xfail.py::TestStuff::test_fail FAILED                                                                                      [ 66%]
test_flaky_xfail.py::TestStuff::test_xfail XFAIL                                                                                      [100%]
test_flaky_xfail.py::TestStuff::test_xfail XFAIL                                                                                      [100%]
test_flaky_xfail.py::TestStuff::test_xfail XFAIL                                                                                      [100%]

================================================================= FAILURES ==================================================================
____________________________________________________________ TestStuff.test_fail ____________________________________________________________

self = <test_flaky_xfail.TestStuff object at 0x111efaca0>

    def test_fail(self):
        # Expect to run three times and fail
>       assert False
E       assert False

test_flaky_xfail.py:13: AssertionError
===Flaky Test Report===

test_ok passed 1 out of the required 1 times. Success!
test_fail failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /Users/hugo/github/pylast/test_flaky_xfail.py:13>]
test_fail failed (1 runs remaining out of 3).
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /Users/hugo/github/pylast/test_flaky_xfail.py:13>]
test_fail failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /Users/hugo/github/pylast/test_flaky_xfail.py:13>]
test_xfail failed (2 runs remaining out of 3).
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /Users/hugo/github/pylast/test_flaky_xfail.py:18>]
test_xfail failed (1 runs remaining out of 3).
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /Users/hugo/github/pylast/test_flaky_xfail.py:18>]
test_xfail failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>
        assert False
        [<TracebackEntry /Users/hugo/github/pylast/test_flaky_xfail.py:18>]

===End Flaky Test Report===
========================================================== short test summary info ==========================================================
FAILED test_flaky_xfail.py::TestStuff::test_fail - assert False
================================================== 1 failed, 1 passed, 3 xfailed in 0.27s ===================================================

Is there a way for me to tell flaky NOT to re-run @pytest.mark.xfail tests?

I'd prefer to have this as a global thing, rather than adding a second decorator to individual tests, if possible.

Thanks!

[Jeff-Meadows]

I think something like the following would work.

def _no_xfail_rerun_filter(err, name, test, plugin):
    for marker in test.iter_markers(name='xfail'):
        return False

@flaky(max_runs=3, min_passes=1, rerun_filter=_no_xfail_rerun_filter)
    # ...

I think you could then just define your own decorator which always passed that rerun filter.

[hugovk]

That works, thank you!

[BoboTiG]

Do you are aware of a global solution? I am using command line arguments passed to pytest to enable flaky, and so I cannot use a custom function to filter out XFailed test.