search

box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
378 stars 60 forks source link

Flaky, xdist, and fixture exceptions can result in test suites running indefinitely #117

Open dshuga opened 7 years ago

dshuga commented 7 years ago

This happens if and only if all of the following are true:

Sample test script:

import pytest
import flaky

@pytest.fixture
def fixture():
    raise AssertionError

@pytest.mark.usefixtures('fixture')
@flaky.flaky(max_runs=2, min_passes=1)
class TestCase:

    @pytest.mark.foobar
    def test_foo(self):
        pass

    def test_bar(self,):
        pass

    def test_baz(self):
        pass

Sample output:

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ pytest -v -n1 test_example.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.5.1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /Users/dshuga/.pyenv/versions/shield/bin/python
cachedir: .cache
rootdir: /Users/dshuga/indeed/shield/products/examples, inifile: 
plugins: indeed-shield-0.0.0, flaky-3.3.0, warnings-0.1.0, xdist-1.15.0
[gw0] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw0] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
gw0 [3]
scheduling tests via LoadScheduling

test_example.py::TestCase::test_foo

(stays this way indefinitely)

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ pytest -v -n3 test_example.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.5.1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /Users/dshuga/.pyenv/versions/shield/bin/python
cachedir: .cache
rootdir: /Users/dshuga/indeed/shield/products/examples, inifile: 
plugins: indeed-shield-0.0.0, flaky-3.3.0, warnings-0.1.0, xdist-1.15.0
[gw0] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw1] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw2] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw0] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
[gw1] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
[gw2] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
gw0 [3] / gw1 [3] / gw2 [3]
scheduling tests via LoadScheduling

test_example.py::TestCase::test_baz 
test_example.py::TestCase::test_bar 
test_example.py::TestCase::test_foo 
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/main.py", line 98, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/main.py", line 133, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/shield/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614, in execute
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/3.5.1/envs/shield/lib/python3.5/site-packages/xdist/dsession.py", line 536, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/3.5.1/envs/shield/lib/python3.5/site-packages/xdist/dsession.py", line 555, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>   File "/Users/dshuga/.pyenv/versions/3.5.1/envs/shield/lib/python3.5/site-packages/xdist/dsession.py", line 593, in slave_slavefinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('test_example.py::TestCase::()::test_bar', <SlaveController gw1>)
INTERNALERROR> assert not 'test_example.py::TestCase::()::test_bar'

============================================================== no tests ran in 1.39 seconds ==============================================================

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $
[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $ pytest -v -n1 -p no:flaky test_example.py
================================================================== test session starts ===================================================================
platform darwin -- Python 3.5.1, pytest-3.0.5, py-1.4.31, pluggy-0.4.0 -- /Users/dshuga/.pyenv/versions/shield/bin/python
cachedir: .cache
rootdir: /Users/dshuga/indeed/shield/products/examples, inifile: 
plugins: indeed-shield-0.0.0, warnings-0.1.0, xdist-1.15.0
[gw0] darwin Python 3.5.1 cwd: /Users/dshuga/indeed/shield/products/examples
[gw0] Python 3.5.1 (default, May  4 2016, 14:23:33)  -- [GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)]
gw0 [3]
scheduling tests via LoadScheduling

test_example.py::TestCase::test_foo 
[gw0] ERROR test_example.py::TestCase::test_foo 
test_example.py::TestCase::test_bar 
[gw0] ERROR test_example.py::TestCase::test_bar 
test_example.py::TestCase::test_baz 
[gw0] ERROR test_example.py::TestCase::test_baz 

========================================================================= ERRORS =========================================================================
__________________________________________________________ ERROR at setup of TestCase.test_foo ___________________________________________________________
[gw0] darwin -- Python 3.5.1 /Users/dshuga/.pyenv/versions/shield/bin/python
@pytest.fixture
    def navigate_to_indeed():
>       raise AssertionError
E       AssertionError

test_example.py:14: AssertionError
__________________________________________________________ ERROR at setup of TestCase.test_bar ___________________________________________________________
[gw0] darwin -- Python 3.5.1 /Users/dshuga/.pyenv/versions/shield/bin/python
@pytest.fixture
    def navigate_to_indeed():
>       raise AssertionError
E       AssertionError

test_example.py:14: AssertionError
__________________________________________________________ ERROR at setup of TestCase.test_baz ___________________________________________________________
[gw0] darwin -- Python 3.5.1 /Users/dshuga/.pyenv/versions/shield/bin/python
@pytest.fixture
    def navigate_to_indeed():
>       raise AssertionError
E       AssertionError

test_example.py:14: AssertionError
================================================================ 3 error in 1.49 seconds =================================================================

[dshuga@i-110516 ~/indeed/shield/products/examples master]
(shield) $

Probably the same root cause as https://github.com/box/flaky/issues/114

jakeczyz commented 7 years ago

Also seeing this. I'm running: platform linux -- Python 3.6.0, pytest-3.0.7, py-1.4.33, pluggy-0.4.0 metadata: {'Python': '3.6.0', 'Platform': 'Linux-4.4.41-k8s-x86_64-with', 'Packages': {'pytest': '3.0.7', 'py': '1.4.33', 'pluggy': '0.4.0'}, 'Plugins': {'xdist': '1.15.0', 'tap': '2.0', 'metadata': '1.3.0', 'html': '1.14.2', 'flaky': '3.3.0'}}