Feature request: ability to set maxRetries at test class and method level

[itkhanz]

Hi, Currently i am using this plugin to retry flaky tests, however I have run into a scenario where i would like to set a custom maxRetries for some specific test class/method to have a different number than the number that gets applied to whole test suite.

This is crucial for our use case, because increasing the number of maxRetries at suite level will mean that all the tests will be retried and this is what we do not want as it can cause unnecessary retries for other tests.

test {
    retry {
        maxRetries = 2
        maxFailures = 20
        failOnPassedAfterRetry = true
    }
}

If it is not possible with the current setup, could you please add this feature to allow users to pass different number of maxRetries to some test methods so that i can run a particular test class/method to either have more or less maxRetries than the default ones? Perhaps it is possible to have this functionality to allow passing maxRerites in filter or classRetry as

test {
    retry {
        maxRetries = 2 //default number of retries will be 2
        maxFailures = 20
        failOnPassedAfterRetry = true
         classRetry / filter{
            maxRetries = 4 //classes define under this section will have maxRetries set to 4 which overrides the default value of 2

            includeClasses.add("*IntegrationTest")
            includeAnnotationClasses.add("*Retryable")
        }
    }
}

Thank you.

[pshevche]

because increasing the number of maxRetries at suite level will mean that all the tests will be retried

Could you please elaborate on that? maxRetries applies only to the failed tests or tests that, for some reason, depend on each other. So, we should not be retrying all tests in the suite, but only those.

In that case, I don't see how a test-specific maxRetries would help. Let's say we have a class TestA that has two methods test1 and test2. For some reason, we detect that the entire class has to be retried if one of the methods fails. In this case, regardless of where you set maxRetries, we will retry the entire class.

I think, the request makes sense, but I just want to confirm that we are not missing some underlying issue.

[itkhanz]

Could you please elaborate on that? maxRetries applies only to the failed tests or tests that, for some reason, depend on each other. So, we should not be retrying all tests in the suite, but only those.

Yeah sorry what I meant by this is that, if we increase the number of maxRetries then it will mean that all the failed tests will be retried with higher retry count.

Lets say I have a ClassA and ClassB, and with maxRetries set to 2, this will retry both these classes 2 times on failure. In some scenarios, lets say ClassB behaviour is too flaky because of some external factors and we need to retry this particular class 5 times instead of 2. Now if i increase the maxRetries to 5, this will also retry ClassA 5 times as well which is what i do not want or expect.

You are right that if we increase the maxRetries at class level, it would then mean retrying of the entire class which could possibly contain multiple test methods. that is why i suggested to have a more better control to filter and set maxRetries at method level. I am not sure if this is possible to apply maxRetries filter at method level.

In any case, test class/method specific maxRetries would help us to specify customised number of maxRetries for our common cases which will then only run that filtered class/method with custom maxRetries instead of changing the retry count for all the failed tests. In other way, this would also be helpful if we do not want to retry a specific test class/method on failure so we can then set the maxRetries count for that specific test to zero.

[pshevche]

I understand; thank you for clarifying this. This makes sense to me, and we will consider this request when we plan the next feature release of the plugin.