Cucumber html report in Jenkins is not generated

[neo81-83]

I'm using latest version of your JVM with Java 21. I've some long running suites which contains 1000+ tests that run for 4-5 hrs with 25 parallel threads. All the tests run successfully but the Cucumber html report is not getting generated in Jenkins. The error is given below:

[INFO] Running courgette.api.junit.Courgette 2024-06-03 15:48:01.940 [ERROR] Test mechanism :: Cannot invoke "org.junit.runner.Description.getTestClass()" because "description" is null 2024-06-03 15:48:03.887 [INFO] 2024-06-03 15:48:04.521 [INFO] Results: 2024-06-03 15:48:04.523 [INFO] 2024-06-03 15:48:04.525 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 2024-06-03 15:48:04.527 [INFO] 2024-06-03 15:48:04.529 [INFO] ------------------------------------------------------------------------ 2024-06-03 15:48:04.531 [INFO] BUILD FAILURE 2024-06-03 15:48:04.533 [INFO] ------------------------------------------------------------------------ 2024-06-03 15:48:04.535 [INFO] Total time: 03:07 h 2024-06-03 15:48:04.536 [INFO] Finished at: 2024-06-03T19:48:03Z 2024-06-03 15:48:04.538 [INFO] Final Memory: 155M/1504M 2024-06-03 15:48:04.540 [INFO] ------------------------------------------------------------------------ 2024-06-03 15:48:04.541 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project automation: 2024-06-03 15:48:04.543 [ERROR] 2024-06-03 15:48:04.545 [ERROR] Please refer to /var/lib/jenkins/workspace/Testing/by~type/regression/Regression-NewRisk/target/surefire-reports for the individual test results. 2024-06-03 15:48:04.546 [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. 2024-06-03 15:48:04.549 [ERROR] There was an error in the forked process 2024-06-03 15:48:04.551 [ERROR] Test mechanism :: Cannot invoke "org.junit.runner.Description.getTestClass()" because "description" is null 2024-06-03 15:48:04.552 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process 2024-06-03 15:48:04.554 [ERROR] Test mechanism :: Cannot invoke "org.junit.runner.Description.getTestClass()" because "description" is null

POM snippet is given below:

`org.apache.maven.plugins

maven-surefire-plugin

            <version>3.1.2</version>
            <configuration>
                <useSystemClassLoader>false</useSystemClassLoader>
                <includes>
                    <include>**/cucumber/runner/TestScenarioRunner.class</include>
                </includes>
                <forkNode implementation="org.apache.maven.plugin.surefire.extensions.SurefireForkNodeFactory"/>
           </configuration>
            <dependencies>
                <dependency>
                    <groupId>org.apache.maven.surefire</groupId>
                    <artifactId>surefire-junit47</artifactId>
                    <version>3.1.2</version>
                </dependency>
            </dependencies>`

What could be the reason? Quick reply would be appreciated.

[neo81-83]

Entire exception:

2024-06-03 20:11:19.160 [INFO] Running courgette.api.junit.Courgette 2024-06-03 20:11:19.162 [ERROR] Test mechanism :: Cannot invoke "org.junit.runner.Description.getTestClass()" because "description" is null 2024-06-03 20:11:19.163 [INFO] 2024-06-03 20:11:19.165 [INFO] Results: 2024-06-03 20:11:19.166 [INFO] 2024-06-03 20:11:19.168 [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 2024-06-03 20:11:19.170 [INFO] 2024-06-03 20:11:19.171 [INFO] ------------------------------------------------------------------------ 2024-06-03 20:11:19.173 [INFO] BUILD FAILURE 2024-06-03 20:11:19.175 [INFO] ------------------------------------------------------------------------ 2024-06-03 20:11:19.177 [INFO] Total time: 53:22 min 2024-06-03 20:11:19.178 [INFO] Finished at: 2024-06-04T00:11:16Z 2024-06-03 20:11:19.180 [INFO] Final Memory: 150M/1504M 2024-06-03 20:11:19.182 [INFO] ------------------------------------------------------------------------ 2024-06-03 20:11:19.184 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:3.1.2:test (default-test) on project automation: 2024-06-03 20:11:19.186 [ERROR] 2024-06-03 20:11:19.187 [ERROR] Please refer to /var/lib/jenkins/workspace/Testing/by~type/regression/Regression/target/surefire-reports for the individual test results. 2024-06-03 20:11:19.189 [ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream. 2024-06-03 20:11:19.190 [ERROR] There was an error in the forked process 2024-06-03 20:11:19.192 [ERROR] Test mechanism :: Cannot invoke "org.junit.runner.Description.getTestClass()" because "description" is null 2024-06-03 20:11:19.194 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process 2024-06-03 20:11:19.196 [ERROR] Test mechanism :: Cannot invoke "org.junit.runner.Description.getTestClass()" because "description" is null 2024-06-03 20:11:19.197 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:628) 2024-06-03 20:11:19.199 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:285) 2024-06-03 20:11:19.201 [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:250) 2024-06-03 20:11:19.203 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1203) 2024-06-03 20:11:19.204 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1055) 2024-06-03 20:11:19.206 [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:871) 2024-06-03 20:11:19.208 [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) 2024-06-03 20:11:19.209 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) 2024-06-03 20:11:19.211 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154) 2024-06-03 20:11:19.213 [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146) 2024-06-03 20:11:19.214 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) 2024-06-03 20:11:19.216 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) 2024-06-03 20:11:19.218 [ERROR] at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) 2024-06-03 20:11:19.220 [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) 2024-06-03 20:11:19.221 [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309) 2024-06-03 20:11:19.225 [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194) 2024-06-03 20:11:19.227 [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107) 2024-06-03 20:11:19.228 [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:955) 2024-06-03 20:11:19.230 [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290) 2024-06-03 20:11:19.232 [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:194) 2024-06-03 20:11:19.233 [ERROR] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) 2024-06-03 20:11:19.235 [ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:580) 2024-06-03 20:11:19.237 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) 2024-06-03 20:11:19.238 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) 2024-06-03 20:11:19.240 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) 2024-06-03 20:11:19.241 [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) 2024-06-03 20:11:19.243 [ERROR] 2024-06-03 20:11:19.245 [ERROR] -> [Help 1] 2024-06-03 20:11:19.247 [ERROR] 2024-06-03 20:11:19.248 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 2024-06-03 20:11:19.250 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 2024-06-03 20:11:19.252 [ERROR] 2024-06-03 20:11:19.253 [ERROR] For more information about the errors and possible solutions, please read the following articles: 2024-06-03 20:11:19.266 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[prashant-ramcharan]

Your surefire plugin returned an error and likely killed the process before Courgette could do any processing.

This is outside the control of Courgette.

 org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process

• How do you know all the tests ran successfully? Do you have any logs at the end of the run?

• Is the Courgette html report generated?

• Does this run locally without issues?

[neo81-83]

Yes, all tests run successfully which is evident from the console logs:

─────────────────────────────────────────────────── 2024-06-04 15:28:13.391 Courgette Test Statistics
2024-06-04 15:28:13.391 ─────────────────────────────────────────────────── 2024-06-04 15:28:13.391 Summary: 84% passed, 16% failed 2024-06-04 15:28:13.391 Duration: 152 min, 13 sec 2024-06-04 15:28:13.391 Run Level: Scenario 2024-06-04 15:28:13.391 Total: 1212 2024-06-04 15:28:13.391 Passed: 1015 2024-06-04 15:28:13.391 Failed: 197 2024-06-04 15:28:13.391 ───────────────────────────────────────────────────

Courgette HTML report is not generated and the folder is empty. There are no issues locally. Can you take a look at my TestRunner class and let me know what could be wrong?

@RunWith(Courgette.class) @CourgetteOptions( threads = 10, runLevel = CourgetteRunLevel.SCENARIO, rerunFailedScenarios = false, testOutput = CourgetteTestOutput.CONSOLE, reportTargetDir = "target", cucumberOptions = @CucumberOptions( features = "src/test/resources/features", glue = "com.xyz.cucumber.stepdefs", tags = "@tag", publish = false, plugin = { "pretty", "json:target/cucumber-report/cucumber.json", "html:target/cucumber-report/cucumber.html"} ))

public class TestScenarioRunner { }

I've already posted pom.xml is my original msg. Also , does courgette.api.junit.Courgette class runs in a separate thread? Any memory specs. required for long running suite?

[krnnarayan]

Hi @prashant-ramcharan , We have a found a solution for this by implementing a change which you had added after our discussion 2 years back on this report generation issue, Just attaching the mail for INFO, where we had discussed. Help in courgette-jvm _ GetSessionID Method access.zip

As part of this discussion you had introduced these options : So we have now used this options, before we couldnt use this option as we were not able to take the latest courgette versions, but we have upgraded out Java to 21 and also we have taken latest courgette version.

Just wanted to request you, now e have close to 32k+ Cases running (During our discussion it was only 16K), so please refrain from removing this options in the upcoming versions , we are now completely dependent on this option as results we are not generating after the runs, we are going to use all the Jsons and nd jsons present in in target/session/ folders.

[krnnarayan]

And also @prashant-ramcharan we saw that, after implementing the above change, we have seen that there are empty Jsons and nd Jsons file, now this is strange, when does this happen any idea ? Is it like the test is skipped or something ?

Just attaching the screenshots from the job for reference where you can see the files are generated with 0 Bytes.

And also we saw that, there is a difference in the number of scenarios run in the courgette INFO which it posts at the end of job. I have attached a jobs details, which is ideally run in the matter of 3 days, and you can see that , some of the tests are skipped, now this is risky sometimes, as the skipped scenario can be found by customer as these scenarios are not tested again my manual team.

Now @prashant-ramcharan do you think this blank json and nd json are related to this skipped cases ? Please do let us know

[Arivoli-cloud]

Hi @prashant-ramcharan , We have a found a solution for this by implementing a change which you had added after our discussion 2 years back on this report generation issue, Just attaching the mail for INFO, where we had discussed. Help in courgette-jvm _ GetSessionID Method access.zip

As part of this discussion you had introduced these options : So we have now used this options, before we couldnt use this option as we were not able to take the latest courgette versions, but we have upgraded out Java to 21 and also we have taken latest courgette version.

Just wanted to request you, now e have close to 32k+ Cases running (During our discussion it was only 16K), so please refrain from removing this options in the upcoming versions , we are now completely dependent on this option as results we are not generating after the runs, we are going to use all the Jsons and nd jsons present in in target/session/ folders.

@krnnarayan is it working now? we are not able to get the report so far. we are planning to execute it with JDK 17. can you please guide us

[prashant-ramcharan]

Those empty reports are unlikely due to skipped tests because skipped tests will also be reported.

I plan on optimising the report processing in a future Courgette release.

This should improve memory / resource usage when Courgette processes multiple Cucumber reports (1k+) into a single report.

[krnnarayan]

Thanks @prashant-ramcharan for the Reply. Yes, Optimizing the report processing will help for sure. Please do notify us once this particular courgette version is released, so that we can try it out.

On Skipped tests part : Can we have some mechanism to add in the courgette info console only the skipped cases, if not then something else which can help team to know , that these were the Skipped cases, might we can print the tags which are present in the skipped scenario, so that team can rerun locally and no issues of tests getting untested and going to production. Please do let us know on your thoughts for the same @prashant-ramcharan.

For now what we have done is , we are deleting the Empty .Json and .NdJson before the jenkins cucumber report plugin runs on this target folder, but this doesnt pick the skipped one's now, so no way team can know what are those cases.

[prashant-ramcharan]

Courgette does not track skipped tests and cannot output the number of skipped tests in a run.

Skipped tests are determined by Cucumber and shown on Cucumber reports.

[krnnarayan]

Thanks @prashant-ramcharan for the quick reply again.

@prashant-ramcharan one quick Question i have if you can help me here understand, these skip tests will still go through After Test rite ? or it doesnt run that also, why i am asking, if its running the Aftertest for skipped cases also, then we can build something in place to fetch the tags for the skipped cases, this will also help the team to find the cases and then they can run those separately.

[prashant-ramcharan]

That's correct. You will have access to the Cucumber before and after hooks in your tests.

[prashant-ramcharan]

I reimplemented the reporting processing in version 6.15.0.

[krnnarayan]

@prashant-ramcharan when we were trying the POC to get the skipped scenarios as we had discussed before, we see lot of variance in every run, we were thinking of this below discussed design , but looks like this Instance ID which is getting appended in .Json and .ndJson is not public static and its private.

What we were planning is, we can store all the Instance IDs and corresponding Scenario tags in once place, and then once the execution is complete, we can scan through the jsons which are 0 Bytes, which actually means skipped, and we could iterate from our list with this ID and get it printed in console for team, the scenario tags which got skipped. We already have written a code in pipeline to delete 0B file before cucumber generation, we were thinking of adding this logic also there only.

this is the one thats appending rite in .Json and .ndjson file rite in target/seesion-reports/ @prashant-ramcharan ??

Can u please provide access to Instance ID so that we can execute plan to get skipped cases, else we dont have any option to find skipped cases.

[prashant-ramcharan]

This is not in scope at the moment.

Did you try using the latest version to determine if the original issue is actually resolved and that the reports are getting generated without issues?

[prashant-ramcharan]

Additionally, why do you need to run all tests to determine the ones that are skipped?

Are your tests being conditionally skipped in your code or are you tagging your tests as skipped?

If you are conditionally skipping your tests in code, why don't you just write the file name or scenario name to a local file when it's skipped?

Not sure what benefit you'd get from having to analyse report files at the end of a test run.

[krnnarayan]

@prashant-ramcharan we have not used the new courgette version, as this issue of some of the cases randomly getting skipped will not be solved even if we take the latest, as its been getting skipped before versions of courgette also. Same point i had discussed here : https://github.com/prashant-ramcharan/courgette-jvm/issues/407#issuecomment-2158249794 You mentioned that, there is no way to find the skipped cases from courgette, but if i have to find the skipped cases and inform the team to rerun those and be safe about all cases getting run atleast once before the release. As you had mentioned i have to do something in our suite level to get the skipped cases, i thought, this Instance ID which is getting appended will help me to do a mapping with the scenario level tags. So the log file will look like this : Instance ID Number of Json : @Module-TC1-123, @Regression Instance ID Number of Json : @Module-TC1-453, @Regression Instance ID Number of Json : @Module-TC1-653, @Regression Now i can compare this list for example with the json which are empty (0B) and get the list.

@prashant-ramcharan if there is any other way i can get the skipped cases details, that will also help.

[prashant-ramcharan]

Just to reiterate, Courgette does not skip tests so it's not an issue that I can solve here.

As previously suggested, could you try with the latest Courgette version?

You may not see reports with 0kb and as such won't have any skipped tests.