search

serenity-bdd / serenity-core

Serenity BDD is a test automation library designed to make writing automated acceptance tests easier, and more fun.
http://serenity-bdd.info
Other
724 stars 518 forks source link

serenity-maven-plugin:3.9.7:aggregate taking more than 30mints to generate report #3253

Open amiteshri92 opened 1 year ago

amiteshri92 commented 1 year ago

What happened?

I was using serenity 2.3.0 version before and report generation was fine, had to upgrade it to 3.9.7 and now the report generation after successful execution of test scenarios is taking more than 30mints... Using- Cucumber+Serenity+Junit

Logs: 1 Scenarios (1 passed) 155 Steps (155 passed) 15m9.611s INFO 2023-09-06 13:05:27,654 [main] com.xxx.qe.ui.runner.BaseCuke - Executing close all...... INFO 2023-09-06 13:05:27,657 [main] com.xxx.qe.ui.core.util.system.PluginUtil - Destroying Framework... INFO 2023-09-06 13:05:28,806 [main] com.xxx.qe.ui.runner.BaseCuke - Got File Name E2E_ProdScripts/7_xxxxxx_E2E_Scenario.feature 2085309 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 933.1 s -- in Parallel01IT 2085921 [INFO] 2085921 [INFO] Results: 2085921 [INFO] 2085921 [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 2085921 [INFO] 2085945 [INFO] 2085945 [INFO] --- serenity-maven-plugin:3.9.7:aggregate (serenity-reports) @ xxx-e2e-ui-plugin-test --- 2113348 [INFO] GENERATING REPORTS USING 32 THREADS 2174982 [INFO]

| TEST NAME: Add promo then go to XXX then Add Frame with Promo code then add components in xxx and complete payment with remake flow | RESULT: Passing | REQUIREMENT: Execution of XXXXXX scneario | REPORT: file:///builds/xxxxxxx-xxxx/xxxx/xxx-qa/xxx-e2e-ui-plugin-test/target/site/serenity/b6bf08d38855b612ef6efb47205d151c_l_7_xxxxxxxx_e2e_scenario.html

2176427 [INFO] GENERATING SUMMARY REPORTS... 2179132 [INFO] GENERATING REQUIREMENTS REPORTS...

and then it stuck at above point for long.

What did you expect to happen?

Report generation should be successful in time.

Serenity BDD version

3.9.7

JDK version

corretto-11

Execution environment

Windows 11 Browser- Chrome Gitlab CI/CD

How to reproduce the bug.

Use any project with above env and take small test to generate report, testcase execution is fine but at the end of aggregate report generation it continues for long.

How can we make it happen?

Add it to the Serenity BDD backlog and wait for a volunteer to pick it up

wakaleo commented 1 year ago

Have you tried with 4.0.1?

jimmykemp commented 1 year ago

We found the same report behaviour as the OP after upgrading from 3.7.1 to 3.9.x as well. Will put it down to our project/enviornment, given the serenity starter projects, using the same serenity versions, don't appear to suffer from the corresponding report generation degradation.

As @wakaleo suggests, upgrading to 4.0.1, is definitely an improvement on reporting execution times, , though also comes with the JDK 11 upgrade. We'll most likely jump from 3.7.1 to 4.0.1.

v3.7.1 also doesn't appear to have the 'EnvironmentVariables' issue 3177, so perhaps a good version to aim for before the jump to 4.0.x.

abuze-tecovin commented 1 year ago

I had the same issue till every version < 4.0.0, but this issues looks to be finally addressed in version 4.0.0

amiteshri92 commented 1 year ago

When I upgrade to 4.0.1, getting below error- D:\UI Automation\xxx-e2e-ui-plugin-test\src\main\java\com\xxx\qe\ui\widgets\LocatorUtil.java:8:39 java: cannot find symbol symbol: class Steps location: package net.thucydides.core.annotations

Package causing th issue it- import net.thucydides.core.annotations.Steps;

wakaleo commented 1 year ago

Read the release notes

amiteshri92 commented 1 year ago

Got it @wakaleo , thanks! upgraded the pkg to - net.thucydides.core.annotations.steps --->net.serenitybdd.annotations.steps

Now getting below erros, looks thucydides still getting reffered from serenity-cucumber6 (v2.6.0 which is latets available)

java.lang.NoClassDefFoundError: net/thucydides/core/ThucydidesSystemProperty

at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
at java.base/java.lang.Class.getConstructor0(Class.java:3342)
at java.base/java.lang.Class.getConstructor(Class.java:2151)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:50)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

Caused by: java.lang.ClassNotFoundException: net.thucydides.core.ThucydidesSystemProperty at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 16 more

wakaleo commented 1 year ago

Serenity has no dependencies on Cucumber 6

amiteshri92 commented 1 year ago

Excluding that causing below issue now-

java.lang.NoClassDefFoundError: io/cucumber/core/runtime/TypeRegistryConfigurerSupplier

at java.base/java.lang.Class.getDeclaredConstructors0(Native Method)
at java.base/java.lang.Class.privateGetDeclaredConstructors(Class.java:3137)
at java.base/java.lang.Class.getConstructor0(Class.java:3342)
at java.base/java.lang.Class.getConstructor(Class.java:2151)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:50)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

Caused by: java.lang.ClassNotFoundException: io.cucumber.core.runtime.TypeRegistryConfigurerSupplier at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 16 more

wakaleo commented 1 year ago

There are all dependency conflicts due to older library versions in your dependencies. Do a dependency analysis and you will find the cause of the errors.

amiteshri92 commented 11 months ago

Here is the dependency chart of serenity and it is referring to serenity-core and serebity-cucumber6 for the above error-

Referred file- net.thucudides.core.steps.service.CleanupMethodAnnotationProvider line inside the file- net.thucydides.core.steps.service.JUnitCleanupMethodAnnotationProvider

And another in cucumber6- image (20)

wakaleo commented 11 months ago

You should not have a dependency on serenity-cucumber6, it should be serenity-cucumber

amiteshri92 commented 11 months ago

Still the same issue is coming- cucumberRemoved Now the issue comes- io.cucumber.core.exception.CucumberException: java.lang.NoClassDefFoundError: io/cucumber/messages/internal/com/google/protobuf/GeneratedMessageV3

at io.cucumber.core.plugin.PluginFactory.newInstance(PluginFactory.java:120)
at io.cucumber.core.plugin.PluginFactory.instantiate(PluginFactory.java:99)
at io.cucumber.core.plugin.PluginFactory.create(PluginFactory.java:63)
at io.cucumber.core.plugin.Plugins.createPlugins(Plugins.java:32)
at io.cucumber.core.plugin.Plugins.<init>(Plugins.java:25)
at io.cucumber.junit.CucumberSerenityBaseRunner.initiatePluginsList(CucumberSerenityBaseRunner.java:84)
at io.cucumber.junit.CucumberSerenityRunner.<init>(CucumberSerenityRunner.java:44)
at net.serenitybdd.cucumber.CucumberWithSerenity.<init>(CucumberWithSerenity.java:12)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:26)
at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:59)
at org.junit.internal.requests.ClassRequest.getRunner(ClassRequest.java:33)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:50)
at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:235)
at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:54)

Caused by: java.lang.NoClassDefFoundError: io/cucumber/messages/internal/com/google/protobuf/GeneratedMessageV3 at com.aventstack.extentreports.cucumber.adapter.ExtentCucumberAdapter.(ExtentCucumberAdapter.java:91) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at io.cucumber.core.plugin.PluginFactory.newInstance(PluginFactory.java:116) ... 23 more Caused by: java.lang.ClassNotFoundException: io.cucumber.messages.internal.com.google.protobuf.GeneratedMessageV3 at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 29 more

Looks like reference to cucumber6 had some meaning, GeneratedMessageV3 is not found.

wakaleo commented 11 months ago

It's still dependency conflicts, not a Serenity issue as such. If you would like specific help to resolve them, you can ask for an ad-hoc support session on https://www.serenity-dojo.com/home1691753380236.

amiteshri92 commented 11 months ago

Oh! that is what i was not aware of these support packages. Thanks for your support so far @wakaleo really appreciate your quick responses. Could you just tell me the reason of not using cucumber6?

wakaleo commented 11 months ago

As of cucumber 7 we switched to serenity-cucumber. The Cucumber APIs are not backward compatible.

On Wed, 13 Dec 2023 at 11:26, Amitesh Srivastava @.***> wrote:

Oh! that is what i was not aware of these support packages. Thanks for your support so far @wakaleo https://github.com/wakaleo really appreciate your quick responses. Could you just tell me the reason of not using cucumber6?

— Reply to this email directly, view it on GitHub https://github.com/serenity-bdd/serenity-core/issues/3253#issuecomment-1853741398, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAANVMVE6MLADCOJGKTIQ6LYJGGHPAVCNFSM6AAAAAA4NLPM42VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJTG42DCMZZHA . You are receiving this because you were mentioned.Message ID: @.***>

amiteshri92 commented 10 months ago

@wakaleo , We found after analysis that serenity.requirements.dir = src/test/resources/features actually scan all the feature files present this path which was around 200 feature files in our case, so serenity aggregate actually scans all the feature files irrespective of which feature files we have run in suite. We were running around 10 feature files only but still it was scanning all 200 files. So we are narrowing this by setting serenity.requirements.dir = src/test/resources/features/folderhavingtenfeaturefiles so that it scans here only than all files, and it significantly reduced our execution time. Please let me know your suggestion on it and also please help to answer why does serenity scan all the files than the files executed?

wakaleo commented 10 months ago

That's the normal bahavior, since the idea is to produce a report that documents all the requirements, not just the ones you execute