Executing tests in parallel

[grahambarrgraham]

Our body of Scala Cucumber BDD tests is growing, and becoming slow, so we are looking at how we might execute them parallel from SBT.

The examples I can find are focused on Java and Maven : e.g. https://github.com/sugatmankar/cucumber-jvm-parallel-plugin-rerun-example. This makes use of a maven code-generator plugin which creates a cucumber-runner class per feature file, along with the sure-fire plugin with a concurrent execution configuration. A report per feature is generated. From what I can make out: cross-talk in glue classes is eliminated using DI of a "scenario scope" entity, which itself is implemented using ThreadLocal. Another alternative is to fork for each feature file, which is probably safer but more heavyweight. And finally, the Cucumber report rendering is not thread-safe, so a final report needs to be stitched together from the individual feature reports.

There seems to be a few areas to solve :

1. How to drive parallel test execution from SBT
2. How to manage "scenario scope" in Scala glue classes (if necessary)
3. How to produce a final report

I may be way off track, so any insight would be help in directing my efforts..

[lewismj]

I've been looking into a few issues, I mentioned on another one, I've been sort of waiting for Cucumber 1.2.6 so that I can get a Scala 2.12 release out and then bundle in a few changes. If it doesn't get released soon I'll stash some changes and go for an intermediate release.

[lewismj]

As per comment on another issue ... I will try to combine this with some other features. Apologies for long wait, there was a big gap in waiting for Scala 2.12 support in Cucumber. The latest snapshots of 2.0.0 (Cucumber) support it, so I've released the existing framework with Scala 2.12 support; so can focus on some updates.

[arpanchaudhury]

@lewismj Can you also update the README on how can we run tests in parallel?

[lewismj]

For running tests in parallel, is this via the runner or plugin? The runner is a Scala test type framework. I think it might be easier to do that than the plugin that creates a JVM. I will take a look in the runner - it should be a case of just using futures or perhaps something nice in Monix for handling this... Will have a look this evening and try to get to this, this week.

[arpanchaudhury]

Awesome. Thanks.

[lewismj]

I will add something into the runner to do 'parallelise individual tests' this should allow someone to switch on a flag that will parallelise each test within a suite. Should be able to do that in the next day or two and try to bundle with a couple of things for a new release this weekend. (fingers crossed).

parallelExecution in Test should run suites in parallel, a separate property can be used to paralellise the individual tests.

[lewismj]

I just looked at this, and in order to parallelize tests, it would mean running multiple Cucumber JVM and stitching together the results as suggested in the first post. A bit more work than I was hoping for.

In the runner it may be possible to parallelize test suites by:

Have a new trait:

trait CucumberTestSuite extends CucumberSpec {
  def features: List[String] = List.empty
}

Allow steps to be defined as:

class AddAndMultiplySteps extends ScalaDsl with EN with Matchers with CucumberTestSuite {
  override def features = List("Multiplication.feature","Addition.feature")

And then in the runner:

    val features = Try {
      val instance = Class.forName(name).newInstance.asInstanceOf[ {def features: List[String]}]
      instance.features
    }.toOption

Which would extract out the features of each suite - which hopefully I can pass as parameters into the Cucumber JVM. Then would allow me to wrap the invokeCucumber in a future and allow switch on of parrallelExecution in Test. It would also be backward compatible, so not immediately break anyone upgrading the runner.

[lewismj]

I just updated the runner, its backward compatible. There is a bit way to run tests in parallel if you mixin a new trait.

At present, I don't stitch together the different outputs, but will look into that in a future release. However, if you have a very large number of tests and want to run them all in parallel, the update will allow that.