Provide a more ergonomic way of automatically retrying failing tests

[pshevche]

Summary

Having the retry(RetryConfig) function is nice, but ideally, I'd like to set the same retry configuration for every single test case in the spec or project. All of my tests are prone to the same infrastructure issue, and should all be retried. Using the retry function in every single one of them adds a lot of boilerplate. Is there a recommended way to set retries globally? I played around with a TestCaseExtension to re-run tests multiple times if the result is failed, but the reporting is a bit odd because intermediate executions are reported as pending.

Thank you for a great framework, and looking forward to your reply!

[pshevche]

If you feel like this could be a fun issue to contribute on, let me know, I'd be happy to take it on.

[sksamuel]

Yeah please raise a PR. Good idea!

[Kantis]

I think you could take some inspiration from how this is done for eventually, here

[pshevche]

Thank you for the pointers, I plan to look into it soon

[pshevche]

I looked into it today. I figured out that what I wanted to achieve required quite a big change. My idea was not just to have a global config for retries (similar to eventually), but also enable retries if it was set. Having a global config seemed pretty straightforward as I could follow other examples like for timeout. But enabling retries based on the config setting seemed pretty invasive and I don't think that the framework does anything like that.

However, I found a way to achieve this with a custom TestCaseExtension. I think it is fine to close the issue.

import io.kotest.assertions.retry
import io.kotest.assertions.retryConfig
import io.kotest.core.extensions.TestCaseExtension
import io.kotest.core.test.TestCase
import io.kotest.core.test.TestResult
import kotlin.time.Duration.Companion.minutes

object RetryExtension : TestCaseExtension {

    val config = retryConfig {
        maxRetry = 2
        timeout = 10.minutes
        exceptionClass = Exception::class
    }

    override suspend fun intercept(testCase: TestCase, execute: suspend (TestCase) -> TestResult): TestResult {
        return execute(testCase.copy(test = {
            retry(config) {
                testCase.test(this)
            }
        }))
    }
}

[Kantis]

@pshevche I think we can leave this issue open for a while. If a lot of people stumble in here looking for answers, we could add something like your extension into the framework perhaps. Let's wait and see.