search

box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
377 stars 60 forks source link

pytest unittest cases are destroyed after first teardown #212

Open Chris-Peterson444 opened 3 months ago

Chris-Peterson444 commented 3 months ago

The following test from test/test_pytest/test_pytest_example.py fails:


@flaky
class TestExampleFlakyTestCase(TestCase):
    _threshold = -1

    @staticmethod
    def test_flaky_thing_that_fails_then_succeeds():
        """
        Flaky will run this test twice.
        It will fail once and then succeed.
        """
        TestExampleFlakyTestCase._threshold += 1
        assert TestExampleFlakyTestCase._threshold >= 1

========================================================================================================================== FAILURES ==========================================================================================================================
_____________________________________________________________________________________________ TestExampleFlakyTestCase.test_flaky_thing_that_fails_then_succeeds _____________________________________________________________________________________________

>       lambda: ihook(item=item, **kwds),  when=when, reraise=reraise
    )
E   AssertionError

flaky/flaky_pytest_plugin.py:148: AssertionError
===Flaky Test Report===

test_flaky_thing_that_fails_then_succeeds failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 >= 1
 +  where 0 = TestExampleFlakyTestCase._threshold
        [<TracebackEntry /usr/lib/python3.12/unittest/case.py:58>, <TracebackEntry /usr/lib/python3.12/unittest/case.py:634>, <TracebackEntry /usr/lib/python3.12/unittest/case.py:589>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/test/test_pytest/test_pytest_example.py:101>]
test_flaky_thing_that_fails_then_succeeds failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>

        [<TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:342>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/flaky/flaky_pytest_plugin.py:148>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_hooks.py:513>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_manager.py:120>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:139>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/threadexception.py:87>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/threadexception.py:63>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:90>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unraisableexception.py:65>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/logging.py:850>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/logging.py:833>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/capture.py:878>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:122>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/skipping.py:257>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/pluggy/_callers.py:103>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:184>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/runner.py:174>, <TracebackEntry /<working-dir>/python-flaky-3.8.1/venv/lib/python3.12/site-packages/_pytest/unittest.py:319>]

===End Flaky Test Report===
================================================================================================================== short test summary info ===================================================================================================================
FAILED test/test_pytest/test_pytest_example.py::TestExampleFlakyTestCase::test_flaky_thing_that_fails_then_succeeds - AssertionError
===================================================================================================================== 1 failed in 0.04s ======================================================================================================================

The first iteration fails (expected) and then each subsequent iteration fails due to an assertion error in pytest: 

    def runtest(self) -> None:
        from _pytest.debugging import maybe_wrap_pytest_function_for_tracing

        testcase = self.instance
>       assert testcase is not None
E       AssertionError

This appears to be due to a bug in pytest itself https://github.com/pytest-dev/pytest/issues/12424