tarantool / test-run

Tarantool functional testing framework
14 stars 17 forks source link

Rerun all failed tests, not only marked as fragile #329

Closed ylobankov closed 2 years ago

ylobankov commented 2 years ago

test-run supports functionality to rerun failed tests in place, but these tests have to be on so called fragile list. To add a test to the fragile list we need to add a special configuration to the suite.ini file of a test suite. Configuration example:

fragile = {
    "retries": 5,
    "tests": {
        "tarantoolctl.test.lua": {
            "issues": [ "gh-5059", "gh-5346" ]
        },
        "debug.test.lua": {
            "issues": [ "gh-5346" ]
        },
        ...
    }
}

Rerunning failed tests in place is quite convenient because it allows us to avoid rerunning all tests again and thus save time.

But to make it work as expected we should keep the list of fragile tests always up-to-date. Flaky tests may be introduced every day and keeping the list of fragile tests always up-to-date becomes extremely difficult to do.

So our solusion is quite simple: just rerun all failed tests. By default, the number of retries for regular and fragile tests is 3. But for fragile tests this number can be overriden in the suite.ini file.

Closes #328