search

serenity-bdd / serenity-jbehave

The Serenity integration for JBehave
39 stars 34 forks source link

Parallel Jbehave Running Test #237

Open jayzsuarez opened 4 years ago

jayzsuarez commented 4 years ago

@wakaleo tried to run in parallel using serenity and jbehave but still run in sequential manner. Already defined thread count and fork count in maven sure fire.

wakaleo commented 4 years ago

Parallel testing with Serenity in JBehave isn't supported due to constraints on the JBehave architecture. I would recommend migrating to Cucumber.

jayzsuarez commented 4 years ago

@wakaleo we have a lot of stories 300+ and we want to slice it per batch :( ~12hrs to finish

wakaleo commented 4 years ago

I would set up separate test runners for each batch and use failsafe to run the test runners in parallel.

jayzsuarez commented 4 years ago

@wakaleo you mean I can define different runners in one command?

e.g. -Dtest_run1, -Dtest_run2

wakaleo commented 4 years ago

No, you would just configure several JUnit classes with different configurations to run different subsets of the stories. It's not a JBehave-specific technique.

zrieq commented 4 years ago

@wakaleo is there a way to run the test runners on different jenkins nodes and merge the reports later ?

To clarify, if I have separated the stories into different subsets using different runners, then I used failsafe to run the test runners in parallel, this will work on a single machine. What about if I want to split the load on different machines using jenkins pipeline and then merge the reports back ?

I have looked at https://serenity-bdd.github.io/theserenitybook/latest/serenity-parallel-batches.html and focused on the cucumber section to run multiple batches and forks in the same time using jenkins. I am trying to do the same stuff using jbehave instead of cucumber (our current project is using Jbehave and it has a lot of stories).

wakaleo commented 4 years ago

Yes, you can use tags, different runner classes, or batches to run a different subset of your tests on each node. Then you need to copy the target directory contents from each node into the same place and run mvn serenity:aggregate

zrieq commented 4 years ago

@wakaleo Thanks for the quick turnaround.

So this means really that I have to take care of the slicing to different machines in the jenkins pipeline script to distribute the load and use the forking in failsafe (to boost the execution on every node) then I will merge back to single place and run mvn serenity:aggregate

it would be nicer if it is supported like in cucumber :)

Thanks Again.

zrieq commented 4 years ago

@wakaleo I have tried to use forking from failsafe plugin, however, it fails after execution with the current error:

 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-failsafe-plugin:2.22.0:verify (verify) on project ngsdf-at-ap: There are test failures.
 [ERROR] 
 [ERROR] Please refer to ~\target\failsafe-reports for the individual test results.
 [ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream.
 [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: ExecutionException There was an error in the forked process
 [ERROR] Test mechanism :: null
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:494)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:368)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:292)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:245)
 [ERROR]    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1194)
 [ERROR]    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1022)
 [ERROR]    at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:868)
 [ERROR]    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
 [ERROR]    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
 [ERROR]    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
 [ERROR]    at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
 [ERROR]    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
 [ERROR]    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
 [ERROR]    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
 [ERROR]    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
 [ERROR]    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
 [ERROR]    at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
 [ERROR]    at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
 [ERROR]    at org.apache.maven.cli.MavenCli.execute(MavenCli.java:954)
 [ERROR]    at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
 [ERROR]    at org.apache.maven.cli.MavenCli.main(MavenCli.java:192)
 [ERROR]    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 [ERROR]    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 [ERROR]    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 [ERROR]    at java.base/java.lang.reflect.Method.invoke(Method.java:567)
 [ERROR]    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
 [ERROR]    at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
 [ERROR]    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
 [ERROR]    at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
 [ERROR] Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: There was an error in the forked process
 [ERROR] Test mechanism :: null
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:658)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:362)
 [ERROR]    at org.apache.maven.plugin.surefire.booterclient.ForkStarter$1.call(ForkStarter.java:342)
 [ERROR]    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 [ERROR]    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
 [ERROR]    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
 [ERROR]    at java.base/java.lang.Thread.run(Thread.java:835)
 [ERROR] -> [Help 1]

After some research, the same issue is encountered in different projects (like cucumber) and it is related to some defect in surefire as per the issue here: https://github.com/cucumber/cucumber-jvm/issues/960

I am not sure if it is the same issue that I am facing or not, but I am sure there is NO parenthesis in stories/scenarios.

Any clue?

P.S.

  1. if I fetch the report, I will find that the serenity report is generated and everything is OK!
  2. There is another issue with the Runners, if I try to move the runners to any package, they will not run in the first place and I will not see the chromedriver working (it sounds like a resolving path issue). No my concern right now.
zrieq commented 4 years ago

Removing the useless config:

<parallel>classes</parallel>

Solved my problem!

It seems JUnit uses different impl once it sees the classes config.

groonyn commented 3 years ago

@wakaleo we have a lot of stories 300+ and we want to slice it per batch :( ~12hrs to finish

@jayzsuarez Hi! I have exactly the same problem with parallelization of 600+ tests written on Serenity+JBehave+Selenium+Java8 . Dificulty is that each scenario is connected with specific credentials on the website, website has a restriction 1 active session per 1 creds. I tried to separate tests by tags of jbehave on 2 groups. I would be very pleased to contact you somehow by any of messengers to discuss that question. my telegram @groonyn. I tried approach with multiconfigurational jenkins job with aggregating serenity report from the each maven sub-build, but reports have some problems, (its a question more to @wakaleo, when aggerate two reports ( cupy -paste into one folder using robocopy script) divided by context, showing ok only main page, but navigation thru test reports in impossible)