search

box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
381 stars 59 forks source link

Output information during the test run #154

Open ulope opened 5 years ago

ulope commented 5 years ago

Depending on test runtime and repetition count flaky tests can take a long time to complete.

It would be nice if there was some feedback on flakyness during the test run, not only afterwards.

For now we're using the following as a slightly hacky workaround but a built in solution would be much nicer:

@pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_call(item):
    """ More feedback for flaky tests.

    In verbose mode this outputs 'FLAKY' every time a test marked as flaky fails.
    This doesn't work under xdist and will therefore show no output.
    """
    outcome = yield
    did_fail = isinstance(outcome._excinfo, tuple) and isinstance(
        outcome._excinfo[1], BaseException
    )
    is_xdist = "PYTEST_XDIST_WORKER" in os.environ
    is_flaky_test = item.get_closest_marker("flaky") is not None
    if did_fail and is_flaky_test and not is_xdist:
        if item.config.option.verbose > 0:
            capmanager = item.config.pluginmanager.getplugin("capturemanager")
            with capmanager.global_and_fixture_disabled():
                item.config.pluginmanager.get_plugin("terminalreporter")._tw.write(
                    f"FLAKY ", yellow=True
                )