box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
378 stars 60 forks source link

Flaky decorators on individual tests have no effect if `--force-flaky` argument is passed #209

Open tdluk opened 4 months ago

tdluk commented 4 months ago

From README.rst:

Pass --max-runs=MAX_RUNS and/or --min-passes=MIN_PASSES to control the behavior of flaky if --force-flaky is specified. Flaky decorators on individual tests will override these defaults.

However, flaky decorators on individual tests do not override these defaults.

To repro the issue, consider the following test file test.py:

import pytest

@pytest.mark.flaky(max_runs=10)
def test_foo():
    assert False

Then run pytest test.py --force-flaky and observe the test was run the default 2 times instead of the 10 specified by the decorator.

tdluk commented 4 months ago

Problem appears to be here:

https://github.com/box/flaky/blob/d29ca76d5a72845428a0716c74f8fa0d40d98532/flaky/flaky_pytest_plugin.py#L252

This means an individual test's decorators won't be translated into flaky attributes, as the test's flaky attributes have already been set to the defaults (or values set in CLI args) here:

https://github.com/box/flaky/blob/d29ca76d5a72845428a0716c74f8fa0d40d98532/flaky/flaky_pytest_plugin.py#L77-L82