only_rerun with exception raised from fixture teardown reruns test but report previous runs as failure

[shlompy]

Hi. I have a need to do some checks in a fixture teardown to determine whether a test should rerun or not. Because rerunfailures only allows filtering based on exception, what I had in mind is that to check whatever I want in the fixture and raise the custom exception so the test will rerun. However, while this package detects a rerun is needed when processing the teardown report, the call (test) report was already processed and it was not marked to be rerun, because the rerun_only condition didn't match the test exception, as the custom exception was raised from the fixture.

Eventually, the test does re-run until it pass, but pytest still report the test failure outcomes.

Note that the reruns works just fine when not using only_rerun condition.

Example code:

import random

import pytest

class MyCustomError(Exception):
    pass

@pytest.fixture
def rerun_custom_checker(request):
    yield

    if request.node.rep_call.outcome == "passed":
        return

    raise MyCustomError()

@pytest.mark.flaky(reruns=10, reruns_delay=1, only_rerun=["MyCustomError"])
def test_example(rerun_custom_checker):

    i = random.randint(2, 4)
    if i != 3:
        assert False

What I had in mind, is to pre-process the reports and set a flag whether rerun is needed, if it was found to be needed for setup/call/teardown. Then, when processing the reports as usual, and just check for this flag.

Something like:

I can raise a PR with that change, assuming this solution is acceptable (It works fine when testing this change)