False scenario skipped output and Exception in version 5.8.1

[pagmoro]

Courgette-JVM Version: 5.8.1

When Test Suite is run individually , gradle reports the Feature as skipped.

2 Scenarios (2 passed)
18 Steps (18 passed)
0m3.461s
automation.RunCucumberTest > courgette.api.junit.Courgette.My Feature Demo SKIPPED

Options:

@RunWith(Courgette.class)
@CourgetteOptions(
        threads = 10,
        runLevel = CourgetteRunLevel.FEATURE,
        rerunFailedScenarios = true,
        rerunAttempts = 1,
        showTestOutput = true,
        reportTargetDir = "build",
        cucumberOptions = @CucumberOptions(
            features = "src/integrationTest/resources/features/",
            glue = {"classpath:com.mysteps", "classpath:com.mypackage"},
            plugin = {"pretty"}
    ))

When run as part of gradle build , build sometimes fails with following Exception

java.lang.NullPointerException
  at courgette.runtime.CourgetteNdJsonCreator.extractTestCaseId(CourgetteNdJsonCreator.java:176)
  at courgette.runtime.CourgetteNdJsonCreator.addMessage(CourgetteNdJsonCreator.java:110)
  at courgette.runtime.CourgetteNdJsonCreator.lambda$getScenarioMessages$1(CourgetteNdJsonCreator.java:77)
  at java.util.ArrayList.forEach(ArrayList.java:1257)
  at courgette.runtime.CourgetteNdJsonCreator.lambda$getScenarioMessages$2(CourgetteNdJsonCreator.java:77)
  at java.util.HashMap.forEach(HashMap.java:1289)
  at courgette.runtime.CourgetteNdJsonCreator.getScenarioMessages(CourgetteNdJsonCreator.java:71)
  at courgette.runtime.CourgetteNdJsonCreator.createScenarioMessages(CourgetteNdJsonCreator.java:55)
  at courgette.runtime.CourgetteReporter.createMessages(CourgetteReporter.java:123)
  at courgette.runtime.CourgetteReporter.<init>(CourgetteReporter.java:33)
  at courgette.runtime.CourgetteRunner.createCucumberReport(CourgetteRunner.java:166)
  at courgette.api.junit.Courgette.run(Courgette.java:53)
  at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
  at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
  at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:43)
  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
  at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
  at java.util.Iterator.forEachRemaining(Iterator.java:116)
  at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
  at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:82)
  at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:73)
  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
  at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
  at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:202)
  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
  at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
  at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:102)
  at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:82)
  at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:78)
  at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  at com.sun.proxy.$Proxy2.stop(Unknown Source)
  at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:132)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:412)
  at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:48)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:56)
  at java.lang.Thread.run(Thread.java:748)

[prashant-ramcharan]

I'm unable to reproduce this.

Can you share a project that I can use to reproduce this?

With regards to the NPE, I can address this in the next release.

[pagmoro]

Thanks for the quick response. I built a sample project , and am only partially able to reproduce it Note that if I revert to 5.8.0 , I do not have any of these issues.

In the Actual Project : 1.) I get the "automation.RunCucumberTest > courgette.api.junit.Courgette.My Feature Demo SKIPPED" whether I run it as a gradle task independently via Command Line or through Intellij's gradle window. 2.) Even if the Test fails , I get the SKIPPED message and Intellij overall status is gray (skipped) 3.) Jenkins run is fine - I do not see the "SKIPPED" message in Jenkins logs

In the sample project: I cannot reproduced the SKIPPED log line. However , intellij still shows gray status and says "Tests ignored" when run with 5.8.1. I'm not sure what is adding the "SKIPPED" message in the actual project logs since in this case Intellij shows ignored even without the Skipped message in the logs.

Since Jenkins is fine , its not a major show stopper , however we cannot use this version as it would throw developers off. Perhaps something introduced in 5.8.1 is not compatible.

Some more info: Intellij version: 2020.3 Gradle version: 5.6.2

[prashant-ramcharan]

Ok, I think I may know why the result is reported as ignored by the JUnit notifier.

Will resolve this in the next release - thanks.

[prashant-ramcharan]

@pagmoro Please try with version 5.9.0. Tests are no longer marked as ignored when using the JUnit runner.

[pagmoro]

Thank you @prashant-ramcharan , I will try this out. Is the NullPointer fixed too? It causes tests to fail sometimes.

[prashant-ramcharan]

Yes, the NPE should be fixed too but let me know if you come across any issues.

[pagmoro]

This one looks good as well - thank you for the quick turnaround!