search

cucumber / cucumber-jvm

Cucumber for the JVM
https://cucumber.io
MIT License
2.7k stars 2.02k forks source link

cucumber junit5 engine doesn't support tests rerun #2805

Open emaks opened 12 months ago

emaks commented 12 months ago

I use cucumber with gradle and run tests with next configuration in build.gradle

dependencies {
    implementation group: 'io.cucumber', name: 'cucumber-junit-platform-engine', version: '7.14.0'
    ...
}
tasks {
    test {
        systemProperties(System.getProperties().sort())
        useJUnitPlatform {
            includeEngines("cucumber")
        }
        systemProperty("cucumber.plugin", "json:build/site/cucumber-report.json, rerun:build/site/failed_scenarios.txt")
        systemProperty("cucumber.features", System.getProperty("folder", "src/test/resources/features"))
        systemProperty("cucumber.glue", "...")
        ...
    }
}

Everting work fine when I just run tests. But if I get "build/site/failed_scenarios.txt" file with failed tests and try to rerun only failed tests with systemProperty("cucumber.features", "@build/site/failed_scenarios.txt") I get an error

org.gradle.api.internal.tasks.testing.TestSuiteExecutionException: Could not complete execution for Gradle Test Executor 7.
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:64)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at com.sun.proxy.$Proxy2.stop(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
Caused by: org.junit.platform.commons.JUnitException: TestEngine with ID 'cucumber' failed to discover tests
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:160)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverSafely(EngineDiscoveryOrchestrator.java:134)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:108)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discover(EngineDiscoveryOrchestrator.java:80)
    at org.junit.platform.launcher.core.DefaultLauncher.discover(DefaultLauncher.java:110)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:119)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:94)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:89)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
    ... 18 more
Caused by: java.lang.IllegalArgumentException: path must exist: D:\tests\@build\site\failed_scenarios.txt
    at io.cucumber.core.resource.PathScanner.findResourcesForPath(PathScanner.java:49)
    at io.cucumber.core.resource.PathScanner.findResourcesForUri(PathScanner.java:31)
    at io.cucumber.core.resource.ResourceScanner.findResourcesForUri(ResourceScanner.java:61)
    at io.cucumber.core.resource.ResourceScanner.scanForResourcesUri(ResourceScanner.java:134)
    at io.cucumber.junit.platform.engine.FeatureResolver.resolveUri(FeatureResolver.java:231)
    at io.cucumber.junit.platform.engine.FeatureResolver.resolveFeatureWithLines(FeatureResolver.java:246)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at io.cucumber.junit.platform.engine.DiscoverySelectorResolver.resolve(DiscoverySelectorResolver.java:71)
    at io.cucumber.junit.platform.engine.DiscoverySelectorResolver.resolveSelectors(DiscoverySelectorResolver.java:48)
    at io.cucumber.junit.platform.engine.CucumberTestEngine.discover(CucumberTestEngine.java:43)
    at org.junit.platform.launcher.core.EngineDiscoveryOrchestrator.discoverEngineRoot(EngineDiscoveryOrchestrator.java:152)
    ... 28 more

with junit4 I used 

task serenityRerun(type: JavaExec) {
    dependsOn(testClasses)
    systemProperties(System.getProperties().sort())
    classpath = sourceSets["test"].runtimeClasspath
    mainClass = 'net.serenitybdd.cucumber.cli.Main'
    args = [
        '--glue', '...',
        '--plugin', '...',
        '@build/site/failed_scenarios.txt'
    ]
}
emaks commented 10 months ago

is there any way to rerun failed tests with junit5?

mpkorstanje commented 10 months ago

To rerun failed scenarios please review https://github.com/cucumber/cucumber-jvm/tree/main/cucumber-junit-platform-engine#rerunning-failed-scenarios.

mpkorstanje commented 10 months ago

While it would be possible to implement a JUnit Engine that executes rerun.txt files, the JUnit 5 Platform does not support ordering test engines. So there is no guarantee that such a cucumber-rerun engine will be invoked after the cucumber engine.

Additionally JUnit 5 somewhat implicitly expects that test execution does not change the tests that could be discovered. While it may be possible with the current state of Surefire and Gradle to ignore that, it could cause all sorts of janky behavior when the JUnit API is used as intended.

More jank than I'm willing to accommodate for.

If this is a major impediment to you, I reckon that you or your company can either petition Gradle and/or Surefire for better JUnit 5 support or materially contribute to that support in some way.

Alternatively, if some one is able to convince the JUnit 5 team to accept a PR that makes it possible order test engines and and willing to commit to dealing with all the jank that comes from implementing cucumber-rerun, I would be willing to accept a PR for that too.

mpkorstanje commented 7 months ago

Additionally JUnit 5 somewhat implicitly expects that test execution does not change the tests that could be discovered.

Potentially implementing TestDescriptor.mayRegisterTests on the rerun engines test descriptor could solve this.

That does leave the test engine ordering problem.

mpkorstanje commented 4 months ago

That does leave the test engine ordering problem.

And the ordering problem could be solved by having multiple surefire executions. One for the run and a second for the rerun. Each targeting different suites. Still janky, but better than nothing.