Run tests in parallel

[lewisob]

As a tester who needs to run lots of tests, it would be of great benefit if the cucumber serenity runner could run tests in parallel (using gradle).

I know there are some other solutions/hacks out there. But none of them are easy to setup and can result in multiple/broken serenity reports.

[wakaleo]

I'm not sure this is doable without rewriting large bits of the Cucumber code base, but I'm open to suggestions or pull requests.

[nbarrett]

The latest cucumber-serenity release(v1.9.16 onwards) now supports parallel running of cucumber tests.

[lewisob]

Hi @nbarrett Is the parallel feature only for CI?

[nbarrett]

@lewisob - not sure what you mean 'only CI', but to date, the only documentation that exists on how to execute tests in parallel is via maven.

[lewisob]

Hi @nbarrett the documentation mentions "when it was run in your CI environment" and "build slaves" a number of times so I thought the instructions may be specific to CI, e.g. a Jenkins build or something.

[nbarrett]

Hi again @lewisob - there's no reason why you can't run all of the batches on different machines without using ci to orchestrate it but it would be a bit more complex to copy all the artifacts manually afterwards - but perfectly possible. All the commands are regular maven commands that can be executed on any machine.

[lewisob]

Thanks @nbarrett Can I run in parallel on 1 machine? Seems like it's still a difficult problem to solve.

[nbarrett]

Yes you can if you use the maven configuration as described in the serenity book. However you need to make sure that your machine has the resources to be able to run the number of browsers specified in the <serenity.fork.count>0${parallel.tests}</serenity.fork.count> property. As a general rule, I wouldn't even recommend doing any parallel testing until you have a test suite that runs reliably (e.g. 100% green) on a single host. Otherwise it can be hard to diagnose problems that can crop up. When running parallel, you will be sharing a single test environment with multiple clients that could easily be accessing/changing shared data that could cause test failures. So you need to be able to distinguish basic test failures from those that are created by multi-users/resource depletion. Once you have got through this, then yes you can run in parallel on one machine by keeping the serenity.batch.count/number=1 and allowing the maven configuration to do the forking for you.

[lewisob]

Ok. I am using Gradle, so will need to convert it. I will attempt. Thanks for your help.

[scormaq]

@lewisob I also use Gradle in my Cucumber-Serenity project. There is no problems with batching, but a bit of problem with forking (I did not found a way to reuse maven-failsafe-plugin mechanism in Gradle build script). Instead of it I managed to deal with forking in Gradle test parallelization mechanism. You can check my Serenity-Cucumber forking solution for Gradle here: https://github.com/scormaq/gradle-serenity-cucumber-forking

[nbarrett]

@lewisob I also use Gradle in my Cucumber-Serenity project. There is no problems with batching, but a bit of problem with forking (I did not found a way to reuse maven-failsafe-plugin mechanism in Gradle build script). Instead of it I managed to deal with forking in Gradle test parallelization mechanism. You can check my Serenity-Cucumber forking solution for Gradle here: https://github.com/scormaq/gradle-serenity-cucumber-forking

This looks awesome @scormaq ! Are you considering creating a PR to add this capability to serenity?

[scormaq]

@nbarrett The only thing that worth to be contributed is SerenityCucumberForker.java, the rest of code is anyway an extra configuration. But I'm not sure that SerenityCucumberForker is implemented in optimal way. I'm doing risky stuff there:

1. I am not 100% sure that gradle worker props always be serial;
2. I'm injecting serenity properties in runtime, but I think this is not the best practice 😄

If you believe that this approach (attaching sliced formatter as cucumber plugin) is acceptable to be merged into main repo, I can propose a PR when I have time.

[mlvandijk]

Note that Cucumber supports parallel execution as of Cucumber-jvm v4.0.0 - see CHANGELOG.md, so this issue is a duplicate of / related to #178

[chronicideas]

@scormaq thanks for sharing your gradle solution much appreciated

[thomas0306]

@scormaq The example code is very helpful since most of the parallel execution examples were done with maven. Thank you!