High memory usage when generating reports with a lot of screenshots

[BminFin]

Hello,

I have been using the Cluecumber reporting plugin (v2.2.0) to generate reports of executed test runs, these runs are scheduled on Jenkins and executed in a seleniumGrid with chromedrivers. Cucumber-java version: 4.8.0

Ever since I increased the amount of scenarios in my test run (205 scenarios now coming from 140, with 15-40 screenshots/scenario) I am getting the following error:

[ERROR] Java heap space -> [Help 1] java.lang.OutOfMemoryError: Java heap space

I will include the full stacktrace at the bottom of this comment. I would also like to point out that I have since reduced the amount of screenshots to 10-15/scenario, but the amount of test scenarios is supposed to more than double in the future as well which would bring the total amount of screenshots embedded within the JSON to about the same amount as it has to work with right now for the current 205 scenarios.

This error is only thrown after every test has finished and when the plugin tries to generate its report based on the information stored within the JSON.

Because of the limited amount of memory available to me and the somewhat large JSON files I am getting at the end of my test runs (200MB for 205 scenario's with screenshots, 6MB without any screenshots) I was wondering just how much memory is actually needed to generate the reports at the end of my test run as I can't tell by looking at the logs.

I have also tried to define the maximum amount of memory to allocate in the heap using the following parameter: -Xmx2048m in MAVEN_OPTS.

plugin [INFO] ------------------------------------------------------------------------ [INFO] Cluecumber Report Maven Plugin, version 2.2.0 [INFO] ------------------------------------------------------------------------ [INFO] - source JSON report directory : /var/jenkins/home/workspace/hellotestproj_TAF/target/cucumber-report/ [INFO] - generated HTML report directory : /var/jenkins/home/workspace/hellotestproj_TAF/target/cluecumber-report [INFO] ------------------------------------------------------------------------ [INFO] - custom parameter : Application -> hellotestproj [INFO] - custom parameter : Environment -> ACC1 [INFO] - custom parameter : CurrentDate -> 2020-01-20T06:36:35Z [INFO] - custom parameter : Empty_Parameter -> null [INFO] ------------------------------------------------------------------------ [INFO] - fail pending/undefined scenarios : false [INFO] - expand before/after hooks : false [INFO] - expand step hooks : false [INFO] - expand doc strings : false [INFO] - page title : HelloTest automation report [INFO] - colors (passed, failed, skipped) : #28a745, #dc3545, #ffc107 [INFO] ------------------------------------------------------------------------ [WARNING] Attempt to (de-)serialize anonymous class org.jfrog.hudson.maven2.MavenDependenciesRecorder$1; see: https://jenkins.io/redirect/serialization-of-anonymous-classes/ [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 04:32 h [INFO] Finished at: 2020-01-20T12:09:02+01:00 [INFO] Final Memory: 450M/1880M [INFO] ------------------------------------------------------------------------ Waiting for Jenkins to finish collecting data [ERROR] Java heap space -> [Help 1] java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf (Arrays.java:3332) at java.lang.StringCoding.safeTrim (StringCoding.java:89) at java.lang.StringCoding.access$100 (StringCoding.java:50) at java.lang.StringCoding$StringDecoder.decode (StringCoding.java:154) at java.lang.StringCoding.decode (StringCoding.java:193) at java.lang.StringCoding.decode (StringCoding.java:254) at java.lang.String. (String.java:546) at java.lang.String. (String.java:566) at com.trivago.cluecumber.filesystem.FileIO.readContentFromFile (FileIO.java:78) at com.trivago.cluecumber.CluecumberReportPlugin.execute (CluecumberReportPlugin.java:108) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:309) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:194) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:107) at org.jvnet.hudson.maven3.launcher.Maven35Launcher.main (Maven35Launcher.java:130) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at jenkins.maven3.agent.Maven35Main.launch (Maven35Main.java:178) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) [ERROR] [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/OutOfMemoryError [JENKINS] Archiving /var/jenkins/home/workspace/hellotestproj_TAF/pom.xml to be.fgov.minfin.HelloTest/HelloTest/0.0.1-SNAPSHOT/HelloTest-0.0.1-SNAPSHOT.pom [JENKINS] Archiving /var/jenkins/home/workspace/hellotestproj_TAF/target/HelloTest-0.0.1-SNAPSHOT.jar to be.fgov.minfin.HelloTest/HelloTest/0.0.1-SNAPSHOT/HelloTest-0.0.1-SNAPSHOT.jar channel stopped [htmlpublisher] Archiving HTML reports... [htmlpublisher] Archiving at BUILD level /var/jenkins/home/workspace/hellotestproj_TAF/target/cluecumber-report to /var/jenkins/home/jobs/hellotestproj_TAF/builds/194/htmlreports/Cluecumber_20Report_20hellotestproj ERROR: Directory '/var/jenkins/home/workspace/hellotestproj_TAF/target/cluecumber-report' exists but failed copying to '/var/jenkins/home/jobs/hellotestproj_TAF/builds/194/htmlreports/Cluecumber_20Report_20hellotestproj'. Finished: FAILURE

[bischoffdev]

Hello @BminFin ,

This is interesting. To be honest, I don't know the exact memory consumption in your case as it is very dependent of the structure/complexity of your json files. Since attachments are huge base64 encoded strings within the json the file size can be quite high. At the moment, Cluecumber consumes the whole json at once and extracts the relevant information. I will look into its streaming functionality (https://sites.google.com/site/gson/streaming) in order to reduce memory consumption. Will add this to the list of improvements.

Higher XMX settings should also solve this - I don't know why it doesn't do it for you :-(

[BminFin]

Appreciate the rapid reply and am looking forward to any future improvements, this really is a great tool when it comes to generating clean reports, sadly the memory usage is holding me back a little. But I can definitely improve some things on my end as well.

Is there any reason why the attachments are not written away/stored elsewhere instead of inside the JSON?

[bischoffdev]

This is just how Cucumber does it. 🤷‍♂

[bischoffdev]

Cluecumber actually converts attachments back to their original format from the JSON Base64 encoded ones so they can be stored in the file system. But it cannot do much about the initial JSON size.

[jaydeepbasu]

Just thinking out loud...@laxersaz can it be solved by embedding screenshot into Json one at a time instead of loading all jsons into memory?

[mpkorstanje]

@BminFin with Cucumber v6 we've implemented a new output format that is streamable and should enable consumers to cope more efficiently with large files. This may, once implemented in cluecumber, resolve this problem.

https://github.com/cucumber/cucumber-jvm/blob/master/release-notes/v6.0.0.md#message-formatter-to-replace-the-json-formatter

However we're also not really aware of a usecase where 15-40 screenshots/scenario is common. A typical usecase would be taking a screenshot when scenario has failed to help debug the problem. Would you mind outlining your motivations for taking this many screenshots?

[BminFin]

Hey,

First of all, thank you for the update!

I work with an application where input makes or breaks the result. Sometimes, looking at the actual result on the final page is not enough, and in those cases being able to go and look at what was filled in on each page, is an easy way of finding any sort of issues. Verifying each input one by one is not really an option as this would increase the runtime even more. This is something I cannot currently do anything about, as parallelization is implemented but I am working with limited resources. There are validations being done on pages prior to the final one, so getting a proper look at those in the form of a screenshot is quite useful. I'd also say that people tend to trust a screenshot more than actual text when it comes to reports.

I'd also like to mention that 15-40 screenshots per scenario was indeed a little excessive, and this was already brought back a significant amount. We are now talking more along the lines of 5-15 screenshots, depending on the type of the scenario.

I hope that clears it up a little. I have been using the one screenshot on failure approach for quite a while now, and while this is usually enough, there are some cases where screenshots of other pages probably could have come in handy.

Kind regards,

Bminfin

[bischoffdev]

By the way, I am currently working on the new version for the Cucumber messaging format.

[shivam9ronaldo7]

Hi @bischoffdev currently I am using 2.8 version of plugin and facing the same issue. Any idea when new version will be launched with fix.

[bischoffdev]

There is currently no new version in progress since the new ndjson message format was not adapted in our test runs. Maybe there will be a new version in the future but at this point, I cannot promise anything.