gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
228 stars 51 forks source link

Retry won't work with Kotlin parametrised tests enclosed in backticks #123

Closed ivans083 closed 7 months ago

ivans083 commented 2 years ago

Sample with Kotlin and JUnit5:

    @MethodSource("anySource")
    @ParameterizedTest
    fun `test method with spaces`(from: Param, to: Param) {
       Assertions.assertTrue(false)
    }

Caused by: java.lang.IllegalStateException: org.gradle.test-retry was unable to retry the following test methods, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues TestsClass#test method with spaces(Param, Param)[1] TestsClass#test method with spaces(Param, Param)[6]

PTParta commented 2 years ago

I'm having the same issue. Has there been progress with this issue? Before I start to change the names of hundreds of test cases it would be nice to know if this has been fixed :)

PTParta commented 2 years ago

Looks like the problem in my case are not the backticks but parentheses in the test's name. For example

@Test fun This is my test (to find bugs)()

won't be retried and gives the Excpetion java.lang.IllegalStateException: org.gradle.test-retry was unable to retry the following test methods, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues

If I remove the parentheses @Test fun This is my test to find bugs()

this works and will be retried.

This bug happened when I added a filter to my gradle test task. Without the filter the tests where retried even with the parentheses in the test names

test{ filter{ excludeTestsMatcthing "a.b.c.*" } }

ivans083 commented 2 years ago

Hey Pauli, I've seen that issue too, there are 2 separate issues:

  1. Backticks on @ParametrizedTest
  2. Parentesis (and I believe maybe some other special chars too) on @Test.

I tested last time a month ago, still not resolved, had to go for renaming test names and avoiding @ParametrizedTest

Cheers,

Ivan.-

On Wed, Mar 2, 2022 at 1:49 AM Pauli Partanen @.***> wrote:

Looks like the problem in my case are not the backticks but parentheses in the test's name. For example

@test https://github.com/test fun This is my test (to find bugs)

won't be retried and gives the Excpetion java.lang.IllegalStateException: org.gradle.test-retry was unable to retry the following test methods, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues

If I remove the parentheses @test https://github.com/test fun This is my test to find bugs

this will be retried.

— Reply to this email directly, view it on GitHub https://github.com/gradle/test-retry-gradle-plugin/issues/123#issuecomment-1056207896, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHZOKKKP73BB2H6S34SV3LU53XMJANCNFSM5JZLPDRA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

Sobolev-Ilya commented 1 year ago

Is there any workarounds for case with parentheses in test method name? (Kotlin + JUnit5) I was trying to use custom DisplayNameGenerator with removing '()' from display test name, but it did not help

Tests with this names wont be retried

@Test
    fun `This test would not retried()`()

@Test
    fun `This test would not retried (too)`()

But this tests will be retried (when paretheses not at the end)

@Test
    fun `This test (would) be retries`()
jean-andre-gauthier commented 7 months ago

This has been fixed in https://github.com/gradle/test-retry-gradle-plugin/pull/272, and will be released in 1.5.8