Incompatible with pytest-timeout plugin

[joaonc]

pytest-rerunfailures is not compatible with the pytest-timeout plugin.

Probably b/c the same pytest hooks are used. The first time, the timeout is observed, but when the test case re-runs, there's no timeout.

pytest-timeoutis more widely used and likely would be easier for the fix to be in this plugin if possible..

Here's a test that fails:

import pytest
pytest_plugins = 'pytester'

def test_pytest_timeout_compatibility_fails(testdir):
    """
    Verifies the `pytest-rerunfailures` plugin is compatible with `pytest-timeout`
    plugin when the test case is not done in the allotted time.
    """
    testdir.makepyfile(
        """
        import time
        import pytest
        @pytest.mark.flaky
        @pytest.mark.timeout(timeout=1)
        def test_times_out():
            time.sleep(2)
            assert True
        """
    )
    result = testdir.runpytest()
    outcome = result.parseoutcomes()
    assert outcome.get('rerun') == 1
    assert outcome.get('failed') == 1

When the test case re-runs, it ignores the timeout and passes.

Note: doesn't matter which decorator comes first (flaky or timeout), the behavior is the same.

[jbms]

The problem is indeed that the same hooks are used. I'm not sure pytest provides a clean way to solve this, other than by combining the two plugins so that the combined functionality can be implemented. However, there is a workaround:

If you run pytest with -o timeout_func_only=true, then the timeout is set in a different hook and is compatible with pytest-rerunfailures. This does change the behavior --- only the main test code will be subject to a timeout, not the setup, but for my purposes it works well enough.

[sallner]

@joaonc It could be possible to tinker with the order in @pytest.hookimpl. The question here would be whether a timeout should include the rerun of a test or not.

[link2xt]

Turns out timeout_func_only=true does not work with timeout_method = thread, but when changed timeout method to default (on Linux), the suggestion to use timeout_func_only=true worked: https://github.com/deltachat/deltachat-core-rust/pull/3064

[jmspereira]

Hey @sallner, do you have any news regarding this issue? pytest-rerunfailures is by far the most maintained pytest extension that does something like this, and it would be very useful if any timeout support would be added...

[triders]

Hi, folks. I've found a workaround, which worked fine for my case and may be helpful for others. I've added the --rerun-except Timeout argument to test runs so that the pytest-rerunfailures won't be triggered when tests have failed because of a global timeout.

addopts = --reruns=2 --rerun-except Timeout --timeout=120

[jmspereira]

Thanks @triders, but at least from my side what I would like to see is the timeout to be considered, not ignored. I.e., if I have a test that fails with timeout but is configured to rerun, I would like that each rerun would consider that timeout