This PR provides native support for running cucumber tests using parallel batches. It offers the following features:
100% deterministic slicing of a Cucumber Serenity test suite into mutually exclusive batches with no configuration or generation of test classes required.
Support for identical system properties as used in the Serenity Junit implementation (e.g. -Dserenity.batch.count=5 -Dserenity.batch.number=4).
Slicing granularity at the scenario level, not the feature level is is the case with other solutions.
Slicing is accomplished by assigning a weight to each scenario where the weight calculation can be based on:
ScenarioLineCountStatistics: the number of steps in each scenario (default).
MultiRunTestStatistics: the historic duration of each scenario (using one or more Serenity results.csv files taken from recent builds as an input). This provides the potential for very accurate slicing, ensuring that all parallel batches have equal run-times and all complete at virtually the same time. Any number of results files can be supplied at the location specified by system property serenity.test.statistics.dir and an average will be calculated.
Fork-based slicing per test execution slave - if your machines are capable of running more than 1 browser at a time, then slicing can be done at the thread level also. To control this, there are two additional system properties serenity.fork.count and serenity.fork.number that can be used in conjunction with the above properties.
No changes are required to the test runner infrastructure, so the same classes annotated with @RunWith(CucumberWithSerenity.class) can still be used.
The test slicing infrastructure infrastructure has been refined over the last 6 months on a client project and produces excellent results on a large business acceptance test suite that historically took over 8 hours to run. However using 25 parallel build slaves (3 of which are each running 8 forks), the run time has now been reduced to under 20 mins (where only 8 minutes of the time is actually spent running tests - the rest is deployment and starting the application stack). Using MultiRunTestStatistics as the slicing algorithm, all slices typically complete within 90 seconds of each other, so there are no 'long tails' delaying report aggregation and parallel job completion.
This PR provides native support for running cucumber tests using parallel batches. It offers the following features:
-Dserenity.batch.count=5 -Dserenity.batch.number=4).ScenarioLineCountStatistics: the number of steps in each scenario (default).MultiRunTestStatistics: the historic duration of each scenario (using one or more Serenity results.csv files taken from recent builds as an input). This provides the potential for very accurate slicing, ensuring that all parallel batches have equal run-times and all complete at virtually the same time. Any number of results files can be supplied at the location specified by system propertyserenity.test.statistics.dirand an average will be calculated.serenity.fork.countandserenity.fork.numberthat can be used in conjunction with the above properties.@RunWith(CucumberWithSerenity.class)can still be used.MultiRunTestStatisticsas the slicing algorithm, all slices typically complete within 90 seconds of each other, so there are no 'long tails' delaying report aggregation and parallel job completion.