search

serenity-bdd / serenity-cucumber-starter

A skeleton project for Serenity BDD and Cucumber JVM
Apache License 2.0
180 stars 285 forks source link

Report genrations taking ages after test execution #60

Closed r2d2-github closed 2 years ago

r2d2-github commented 2 years ago

Hi,

My test suite has:

Whenever I do regression full suite - it takes 5 hr for test execution But 8+ hr to just create the report, the total duration would be 13+ hr.

And during the whole process of test report generation it says WARNING as java.util.concurrent.FutureTask.get(FutureTask.java:205) for various tag names and epic names I am using in my suite.

Following are the some complete WARNING messages:

[INFO] --- serenity-maven-plugin:3.0.5:aggregate (serenity-reports) @ --- Downloading from nexus: https://dps-nexus.service.xxxxx:8443/repository/maven-public/io/cucumber/messages/maven-metadata.xml Downloaded from nexus: https://dps-nexus.service.xxxxx:8443/repository/maven-public/io/cucumber/messages/maven-metadata.xml (2.1 kB at 749 B/s) [WARNING] Report generation timed out for Test Summary Report - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for net.thucydides.core.reports.html.AggregateReportingTask@4424285 - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:SmokeTest - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:EndToEndTest - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tester:ibrahimkhaleel - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:EndToEndTest - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tester:ibrahimkhaleel - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tester:ibrahimkhaleel - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:EndToEndTest - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:RegressionFullSuite - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:RegressionTest - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:RegressionFullSuite - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tester:ibrahimkhaleel - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:RegressionFullSuite - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for TagReportingTask for tag:EndToEndTest - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for ResultReportingTask{reportName='01f5288b80adbd3af52a8ed68a4a616d3164f750229f80da1ef344382d948835.html'} - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for ResultReportingTask{reportName='bde1ac0a2e7f03f0874d567d3aa19f7d580949d77eb11a9d222b6eb1a6fb3a28.html'} - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for ResultReportingTask{reportName='a57c05f0723d2308d06307917c2c8fc04410da8111f010353bea595ac6ee6f5b.html'} - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for ResultReportingTask{reportName='6d87e60bd1c275843692f2b004ff9aafc9c1a10e9237b8a7c6fe5f4eff8974e0.html'} - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for ResultReportingTask{reportName='14f78ec618fa01c03b15cc88ad5c174bdf6d377e4c05592bab00d584c48175ab.html'} - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for ResultReportingTask{reportName='49d119ed65a1eb1d7159226927843f4903c811424197d5de70eef1330843139d.html'} - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for Requirements report capabilities.html - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for net.thucydides.core.reports.html.RequirementsTypeReportingTask@988707cd - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] Report generation timed out for net.thucydides.core.reports.html.RequirementsTypeReportingTask@46bf13a0 - java.util.concurrent.TimeoutException java.util.concurrent.FutureTask.get(FutureTask.java:205) [WARNING] SOME REPORT PAGES COULD NOT BE GENERATED

[INFO] ----------------------------------------- [INFO] SERENITY TESTS: FAILURE [INFO] ----------------------------------------- [INFO] | Tests executed | 316 [INFO] | Tests passed | 309 [INFO] | Tests failed | 4 [INFO] | Tests with errors | 0 [INFO] | Tests compromised | 0 [INFO] | Tests aborted | 0 [INFO] | Tests pending | 3 [INFO] | Tests ignored/skipped | 0 [INFO] ------------------------ | -------------- [INFO] | Total Duration | 4h 55m 46s [INFO] | Fastest test took | 000ms [INFO] | Slowest test took | 3m 59s [INFO] -----------------------------------------

image

And generated report unable to show - Requirements, Epics, Features tabs, it shows the following 404 error.

image image

My current Serenity version is 3.0.5 and Cucumber 6 2.6.0

    <serenity.version>3.0.5</serenity.version>
    <serenity.maven.version>3.0.5</serenity.maven.version>
    <serenity.cucumber.version>2.6.0</serenity.cucumber.version>
    <maven.failsafe-plugin.version>3.0.0-M5</maven.failsafe-plugin.version>
    <maven.surefire-plugin.version>3.0.0-M5</maven.surefire-plugin.version>
    <maven-enforcer-plugin.version>3.0.0</maven-enforcer-plugin.version>
    <maven.compiler-plugin.version>3.8.1</maven.compiler-plugin.version>

Could you please advise the issue here?

wakaleo commented 2 years ago

315 tests is not a lot - it should take minutes to generate the reports. However, 1750+ tags IS a lot. This will create an unresolvable combinatorial explosion of reports that need to be generated which will break the reporting.

Here is what you can try:

r2d2-github commented 2 years ago

Could you please let me know if there is any value we have for serenity.report.exclude.tags property I can exclude all tags I have in my pack.

And 3.1.1 is not working when I include Manual test cases. Also, in report side the graph is not embedded. Currently my test pack in under execution, and I can share more details once this run is finished by evening.

wakaleo commented 2 years ago

Not currently. Could you please raise this issue in serenity-core (this isn't the appropriate repo for it) and clarify what you mean by "not working"?

r2d2-github commented 2 years ago

Sure, I will do that with screenshots after soemtime.

wakaleo commented 2 years ago

If this is important for your company, have you considered getting some commercial support for this?

r2d2-github commented 2 years ago

Hi @wakaleo this is for the NHS, At this moment I am not right person to consider commercial support, but if I am able to elevate this framework as per our needs, we have plans to use it for a wider range. Then I can able to mention that trainings and commercial support you provide.

r2d2-github commented 2 years ago

Hi @wakaleo what is the correct way of defining below property:

serenity.report.exclude.tags="tag1,tag2,tag3"

OR

serenity.report.exclude.tags="@tag1,@tag2,@tag3"

OR

serenity.report.exclude.tags=tag1,tag2,tag3

OR

serenity.report.exclude.tags=@tag1,@tag2,@tag3

I tried all these combinations and they are still visible in my serenity report.

Thank you

wakaleo commented 2 years ago

serenity.report.exclude.tags="tag1,tag2,tag3"

Also try with the hidden.tags property.

r2d2-github commented 2 years ago

serenity.report.exclude.tags="tag1,tag2,tag3" is working with 3.1.1 (not with 3.0.5), However, for the other two issues I have raised at: https://github.com/serenity-bdd/serenity-core/issues/2620 in Serenity Core.

I am going to run my full pack tonight, excluding manual and other tags, then I will let you know how it goes.

wakaleo commented 2 years ago

Closing presumed fixed.