search

gradle / test-retry-gradle-plugin

Gradle plugin to retry tests that have failed to mitigate test flakiness.
Apache License 2.0
213 stars 47 forks source link

Retry plugin fails with kotest framework #93

Open NikolayMetchev opened 3 years ago

NikolayMetchev commented 3 years ago

Using the kotest framework (https://kotest.io/) breaks the retry plugin; Here is the error message:

Execution failed for task ':test'.
> org.gradle.test-retry was unable to retry the following test methods, which is unexpected. Please file a bug report at https://github.com/gradle/test-retry-gradle-plugin/issues
     com.test.KoTest#Test 1

Here is a full project that recreates the failure: kotest.gradle.tar.gz

ldaley commented 3 years ago

@marcphilipp something seems to go wrong in the filtering/selection between Gradle and platform. Could you take a look at kotest when you get a chance please and see if it is implementing this part of platform as we expect.

NikolayMetchev commented 3 years ago

Any news here?

marcphilipp commented 3 years ago

I'm afraid I haven't had time to investigate, yet. It's still on my list, though.

NikolayMetchev commented 2 years ago

Is it still on the list?

marcphilipp commented 2 years ago

Sorry for the delay. I took a look and the problem seems to be that kotest does not use MethodSource which Gradle filters for so even ./gradlew test --tests "KoTest.Test 1" doesn't work. As a stop gap, I think the test-retry plugin could be changed to detect kotest test classes and rerun the entire test class if one of its tests fails.

@NikolayMetchev Would you be interested in contributing a PR for that?

NikolayMetchev commented 2 years ago

I have asked the kotest channel for help: https://kotlinlang.slack.com/archives/CT0G9SD7Z/p1634897392015500

NikolayMetchev commented 2 years ago

Also there is a kotest plugin for Idea. That can run individual tests but it doesn't go through gradle it runs the following. Not sure if that helps.

/Users/nmetchev/.asdf/installs/java/temurin-11.0.12+7/bin/java -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar=52321:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar:/Users/nmetchev/Downloads/kotest-gradle/build/classes/kotlin/test:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-runner-junit5-jvm/4.6.3/8e48bd1180d7bc8c483b61f9531adacfa9dcf919/kotest-runner-junit5-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-core-jvm/4.6.3/685fd4aa66941854e60069429c26245b80e05b87/kotest-assertions-core-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-property-jvm/4.6.3/12a27152669524c9bcd6bcccd82f33f3bdf16b55/kotest-property-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.5.31/ff5d99aecd328872494e8921b72bf6e3af97af3e/kotlin-stdlib-jdk8-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm/1.5.0/d8cebccdcddd029022aa8646a5a953ff88b13ac8/kotlinx-coroutines-core-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-suite-api/1.6.2/36796f8e82db47f0a09580be1fb68449b3b8ee6f/junit-platform-suite-api-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.2/c9ba885abfe975cda123bf6f8f0a69a1b46956d0/junit-jupiter-api-5.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-launcher/1.6.2/d866de2950859ca1c7996351d7b3d97428083cd0/junit-platform-launcher-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.2/1752cad2579e20c2b224602fe846fc660fb35805/junit-platform-engine-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.5.31/77e0f2568912e45d26c31fd417a332458508acdf/kotlin-stdlib-jdk7-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.31/6628d61d0f5603568e72d2d5915d2c034b4f1c55/kotlin-stdlib-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.5.31/43331609c7de811fed085e0dfd150874b157c32/kotlin-stdlib-common-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-engine-jvm/4.6.3/6d6d4b3a3b41125e852e9b50544a6009c9e3ec69/kotest-framework-engine-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-api-jvm/4.6.3/1f8dd5cf0ea1525da216c7daa507c3c2a7c3651e/kotest-framework-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-discovery-jvm/4.6.3/e9e797db0a56fdf317afc906ef731823e8dbb0a2/kotest-framework-discovery-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-common-jvm/4.6.3/180634419642124484a14583fec51bfdbbb44330/kotest-common-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-extensions-jvm/4.6.3/f813249aa945ae1dd17c053c0a6d24f72409386f/kotest-extensions-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-concurrency-jvm/4.6.3/3528b540b65aa1dea63f22b23499ebc2dc6cb77d/kotest-framework-concurrency-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.2/7644a14b329e76b5fe487628b50fb5eab6ba7d26/junit-platform-commons-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-shared-jvm/4.6.3/7e075a5a0dec96d624a165c9ac14af7cccfe455d/kotest-assertions-shared-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/mordant/1.2.1/6cbab1a74ab6dafbf81b7706733d4c2fbaff2e0b/mordant-1.2.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-runtime/1.5.0/a2d66828077376fc1851a6fdbf25865935f4101d/kotlin-script-runtime-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.105/74132d64981decf7c76a506b023e753c335e1983/classgraph-4.8.105.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/colormath/1.2.0/c62f49b31f34588dbbfb477c08fd56bc3026d202/colormath-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-api-jvm/4.6.3/2c4f417ffd5fd8f862ab5250bb68b73f5479f31/kotest-assertions-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.5.0/47806fe8ed30dbdf9e697eda5e9c9a3905ff3363/kotlin-reflect-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-jdk8/1.5.0/ec8255f97a375b6014cad1169c057f98f0ebcf28/kotlinx-coroutines-jdk8-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.java-diff-utils/java-diff-utils/4.9/3ec791c5aa74a72fb499ae8d9547abe27b637b0f/java-diff-utils-4.9.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.mifmif/generex/1.0.2/b378f873b4e8d7616c3d920e2132cb1c87679600/generex-1.0.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/dk.brics.automaton/automaton/1.11-8/6ebfa65eb431ff4b715a23be7a750cbc4cc96d0f/automaton-1.11-8.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk/1.9.3/d26e5a4a7fd4ecb5635b28a0b70960ab0b20556d/mockk-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.6/815893df5f31da2ece4040fe0a12fd44b577afaf/commons-io-2.6.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-util/1.5.0/34f5af4377938350a351eca6b7f9e514388e6aa5/kotlin-script-util-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-jvm/1.5.0/6f2bbad9181cff63b847ac0526cf3d11e5f02f93/kotlin-scripting-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl-jvm/1.9.3/63de994ae1f30b06395325258d2e5683a8bdd525/mockk-dsl-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-common/1.9.3/7f2d6bc3390a76edeba34947e0de6d2314645f25/mockk-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-jvm/1.9.3/d6f7b4d893caf1cda2bb28ee637b247ced5d5eab/mockk-agent-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-daemon-client/1.5.0/373d55b6113e6be520d84bfb738bd40520004bfa/kotlin-daemon-client-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.intellij.deps/trove4j/1.0.20181211/216c2e14b070f334479d800987affe4054cd563f/trove4j-1.0.20181211.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-common/1.5.0/5b78b9cc03552532e8e689a002f1149ab43e44f0/kotlin-scripting-common-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl/1.9.3/4f158b4ed161bc4373356733088e09491ecd13da/mockk-dsl-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-common/1.9.3/50f32c350cb9018a62e6de2bed710acf675f887b/mockk-agent-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-api/1.9.3/467c1fd4867fc49b98a7de421a717b619767abfe/mockk-agent-api-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.objenesis/objenesis/3.0.1/11cfac598df9dc48bb9ed9357ed04212694b7808/objenesis-3.0.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.9.10/211a2b4d3df1eeef2a6cacf78d74a1f725e7a840/byte-buddy-1.9.10.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.9.10/9674aba5ee793e54b864952b001166848da0f26b/byte-buddy-agent-1.9.10.jar io.kotest.engine.launcher.MainKt --spec com.test.KoTest --testpath Test 1 --reporter teamcity
christophsturm commented 2 years ago

I'm the author of a different kotlin test runner,(https://github.com/failgood/failgood) that also uses a dsl instead of methods to declare tests, and I think autodetecting test runners and then changing the behavior is not a great way to do this. @marcphilipp are you suggesting to create methodSources for tests even if they are not declared in a method? would that work?

christophsturm commented 2 years ago

Also there is a kotest plugin for Idea. That can run individual tests but it doesn't go through gradle it runs the following. Not sure if that helps.

/Users/nmetchev/.asdf/installs/java/temurin-11.0.12+7/bin/java -Didea.test.cyclic.buffer.size=1048576 -javaagent:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar=52321:/Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/bin -Dfile.encoding=UTF-8 -classpath /Users/nmetchev/Library/Application Support/JetBrains/Toolbox/apps/IDEA-U/ch-1/212.5457.46/IntelliJ IDEA 2021.2 EAP.app/Contents/lib/idea_rt.jar:/Users/nmetchev/Downloads/kotest-gradle/build/classes/kotlin/test:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-runner-junit5-jvm/4.6.3/8e48bd1180d7bc8c483b61f9531adacfa9dcf919/kotest-runner-junit5-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-core-jvm/4.6.3/685fd4aa66941854e60069429c26245b80e05b87/kotest-assertions-core-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-property-jvm/4.6.3/12a27152669524c9bcd6bcccd82f33f3bdf16b55/kotest-property-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk8/1.5.31/ff5d99aecd328872494e8921b72bf6e3af97af3e/kotlin-stdlib-jdk8-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-core-jvm/1.5.0/d8cebccdcddd029022aa8646a5a953ff88b13ac8/kotlinx-coroutines-core-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-suite-api/1.6.2/36796f8e82db47f0a09580be1fb68449b3b8ee6f/junit-platform-suite-api-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.jupiter/junit-jupiter-api/5.6.2/c9ba885abfe975cda123bf6f8f0a69a1b46956d0/junit-jupiter-api-5.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-launcher/1.6.2/d866de2950859ca1c7996351d7b3d97428083cd0/junit-platform-launcher-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-engine/1.6.2/1752cad2579e20c2b224602fe846fc660fb35805/junit-platform-engine-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-jdk7/1.5.31/77e0f2568912e45d26c31fd417a332458508acdf/kotlin-stdlib-jdk7-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib/1.5.31/6628d61d0f5603568e72d2d5915d2c034b4f1c55/kotlin-stdlib-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-stdlib-common/1.5.31/43331609c7de811fed085e0dfd150874b157c32/kotlin-stdlib-common-1.5.31.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-engine-jvm/4.6.3/6d6d4b3a3b41125e852e9b50544a6009c9e3ec69/kotest-framework-engine-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-api-jvm/4.6.3/1f8dd5cf0ea1525da216c7daa507c3c2a7c3651e/kotest-framework-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-discovery-jvm/4.6.3/e9e797db0a56fdf317afc906ef731823e8dbb0a2/kotest-framework-discovery-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-common-jvm/4.6.3/180634419642124484a14583fec51bfdbbb44330/kotest-common-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-extensions-jvm/4.6.3/f813249aa945ae1dd17c053c0a6d24f72409386f/kotest-extensions-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-framework-concurrency-jvm/4.6.3/3528b540b65aa1dea63f22b23499ebc2dc6cb77d/kotest-framework-concurrency-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.apiguardian/apiguardian-api/1.1.0/fc9dff4bb36d627bdc553de77e1f17efd790876c/apiguardian-api-1.1.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.junit.platform/junit-platform-commons/1.6.2/7644a14b329e76b5fe487628b50fb5eab6ba7d26/junit-platform-commons-1.6.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.opentest4j/opentest4j/1.2.0/28c11eb91f9b6d8e200631d46e20a7f407f2a046/opentest4j-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-shared-jvm/4.6.3/7e075a5a0dec96d624a165c9ac14af7cccfe455d/kotest-assertions-shared-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains/annotations/13.0/919f0dfe192fb4e063e7dacadee7f8bb9a2672a9/annotations-13.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/mordant/1.2.1/6cbab1a74ab6dafbf81b7706733d4c2fbaff2e0b/mordant-1.2.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-runtime/1.5.0/a2d66828077376fc1851a6fdbf25865935f4101d/kotlin-script-runtime-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.classgraph/classgraph/4.8.105/74132d64981decf7c76a506b023e753c335e1983/classgraph-4.8.105.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.ajalt/colormath/1.2.0/c62f49b31f34588dbbfb477c08fd56bc3026d202/colormath-1.2.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.kotest/kotest-assertions-api-jvm/4.6.3/2c4f417ffd5fd8f862ab5250bb68b73f5479f31/kotest-assertions-api-jvm-4.6.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-reflect/1.5.0/47806fe8ed30dbdf9e697eda5e9c9a3905ff3363/kotlin-reflect-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlinx/kotlinx-coroutines-jdk8/1.5.0/ec8255f97a375b6014cad1169c057f98f0ebcf28/kotlinx-coroutines-jdk8-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.github.java-diff-utils/java-diff-utils/4.9/3ec791c5aa74a72fb499ae8d9547abe27b637b0f/java-diff-utils-4.9.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/com.github.mifmif/generex/1.0.2/b378f873b4e8d7616c3d920e2132cb1c87679600/generex-1.0.2.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/dk.brics.automaton/automaton/1.11-8/6ebfa65eb431ff4b715a23be7a750cbc4cc96d0f/automaton-1.11-8.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk/1.9.3/d26e5a4a7fd4ecb5635b28a0b70960ab0b20556d/mockk-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/commons-io/commons-io/2.6/815893df5f31da2ece4040fe0a12fd44b577afaf/commons-io-2.6.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-script-util/1.5.0/34f5af4377938350a351eca6b7f9e514388e6aa5/kotlin-script-util-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-jvm/1.5.0/6f2bbad9181cff63b847ac0526cf3d11e5f02f93/kotlin-scripting-jvm-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl-jvm/1.9.3/63de994ae1f30b06395325258d2e5683a8bdd525/mockk-dsl-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-common/1.9.3/7f2d6bc3390a76edeba34947e0de6d2314645f25/mockk-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-jvm/1.9.3/d6f7b4d893caf1cda2bb28ee637b247ced5d5eab/mockk-agent-jvm-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-daemon-client/1.5.0/373d55b6113e6be520d84bfb738bd40520004bfa/kotlin-daemon-client-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.intellij.deps/trove4j/1.0.20181211/216c2e14b070f334479d800987affe4054cd563f/trove4j-1.0.20181211.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.jetbrains.kotlin/kotlin-scripting-common/1.5.0/5b78b9cc03552532e8e689a002f1149ab43e44f0/kotlin-scripting-common-1.5.0.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-dsl/1.9.3/4f158b4ed161bc4373356733088e09491ecd13da/mockk-dsl-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-common/1.9.3/50f32c350cb9018a62e6de2bed710acf675f887b/mockk-agent-common-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/io.mockk/mockk-agent-api/1.9.3/467c1fd4867fc49b98a7de421a717b619767abfe/mockk-agent-api-1.9.3.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/org.objenesis/objenesis/3.0.1/11cfac598df9dc48bb9ed9357ed04212694b7808/objenesis-3.0.1.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy/1.9.10/211a2b4d3df1eeef2a6cacf78d74a1f725e7a840/byte-buddy-1.9.10.jar:/Users/nmetchev/.gradle/caches/modules-2/files-2.1/net.bytebuddy/byte-buddy-agent/1.9.10/9674aba5ee793e54b864952b001166848da0f26b/byte-buddy-agent-1.9.10.jar io.kotest.engine.launcher.MainKt --spec com.test.KoTest --testpath Test 1 --reporter teamcity

kotest also has a gradle plugin, so it should probably be supported there.

marcphilipp commented 2 years ago

are you suggesting to create methodSources for tests even if they are not declared in a method? would that work?

No, I wasn't suggesting that and it wouldn't help for kotest. The kotest TestEngine implementation currently creates all descriptors dynamically at execution instead of at discovery time and Gradle currently filters the discovery result using a PostDiscoveryFilter.

I think autodetecting test runners and then changing the behavior is not a great way to do this.

If Gradle would somehow support filtering kotest tests that would certainly be preferable. However, I don't see a clear path forward bar reimplementing the kotest TestEngine or adding a different dynamic filtering concept to the JUnit Platform and using it in Gradle. This plugin already has logic for detecting certain Spock classes (e.g. ones annotated with @Stepwise) and rerunning the entire test class. So, while not super elegant, it would provide an interim solution.

christophsturm commented 2 years ago

adding support for tests that are not tied to methods to junit platform would be really great, and imo much better than adding custom behavior for every test engine. isnt the whole point of junit platform not to do that?

marcphilipp commented 2 years ago

The JUnit Platform supports retries via UniqueIdSelector. This is ultimately an issue with Gradle which is very test class/method centric at the moment.

NikolayMetchev commented 2 years ago

The JUnit Platform supports retries via UniqueIdSelector. This is ultimately an issue with Gradle which is very test class/method centric at the moment.

Is it worth opening an issue with Gradle itself?

sksamuel commented 2 years ago

Sorry for the delay. I took a look and the problem seems to be that kotest does not use MethodSource which Gradle filters for so even ./gradlew test --tests "KoTest.Test 1" doesn't work. As a stop gap, I think the test-retry plugin could be changed to detect kotest test classes and rerun the entire test class if one of its tests fails.

@NikolayMetchev Would you be interested in contributing a PR for that?

Kotest will work for --tests if it is a classname only. So the workaround could be as simple as not including the test/method name in the string.

The reason Kotest doesn't support --tests fully is because of the reasons discussed - gradle turns it into a method selector and then doesn't propagate it fully on non method based engines.

christophsturm commented 2 years ago

The JUnit Platform supports retries via UniqueIdSelector. This is ultimately an issue with Gradle which is very test class/method centric at the moment.

Is it worth opening an issue with Gradle itself?

I think it would be a great idea.

eriwen commented 2 years ago

Related to https://github.com/gradle/gradle/pull/19513