`rerun_filter` called twice per test

[ryan-woo]

I am finding that the rerun_filter function is called twice per test.

Version: 3.6.1 Pytest version: 5.4.3

Here is some sample code to recreate the problem:

def my_rerun_filter(*args):
    log.debug("ran filter")
    return True

runs = 0
class TestFlaky:

    @flaky(max_runs=10, rerun_filter=my_rerun_filter)
    def test_4_fails(self):
        global runs
        runs += 1
        log.debug("ran test {}".format(runs))
        if runs < 5:
            raise ValueError()

This logs ran filter 8 times even though the test only runs 4 times. A stack trace shows that the flaky_pytest_plugin.py will call self._should_rerun_test twice: once at line 92 and once at line 101.

Relevant stack traces:

File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 92, in pytest_runtest_protocol
    self.runner.pytest_runtest_protocol(item, nextitem)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/_pytest/runner.py", line 85, in pytest_runtest_protocol
    runtestprotocol(item, nextitem=nextitem)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/_pytest/runner.py", line 100, in runtestprotocol
    reports.append(call_and_report(item, "call", log))
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 141, in call_and_report
    if self._will_handle_test_error_or_failure(item, name, err):
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/_flaky_plugin.py", line 152, in _will_handle_test_error_or_failure
    return self._should_handle_test_error_or_failure(test) and self._should_rerun_test(test, name, err)

File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 101, in pytest_runtest_protocol
    should_rerun = not skipped and self.add_failure(item, call_info.excinfo)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/flaky_pytest_plugin.py", line 327, in add_failure
    return self._handle_test_error_or_failure(item, error)
  File "/Users/rwoo/rwoo/lib/python3.6/site-packages/flaky/_flaky_plugin.py", line 190, in _handle_test_error_or_failure
    if self._should_rerun_test(test, name, err):

[ns-ppawaskar]

Hello, Did you find any resolution to this? Even I am facing similar issue.

[nicki-krizek]

I ran into this as well. It might be useful to at least document this behaviour near the delay_rerun example in README.rst.

[cygnus2048]

Just ran into this also. Any status on the fix?