Enable parallel behat runs

[marxjohnson]

I tried the following:

export MOODLE_DOCKER_WWWROOT=/path/to/moodle/code
export MOODLE_DOCKER_DB=pgsql
export MOODLE_DOCKER_BROWSER=chrome
cp config.docker-template.php $MOODLE_DOCKER_WWWROOT/config.php
bin/moodle-docker-compose up -d
bin/moodle-docker-wait-for-db
bin/moodle-docker-compose run webserver php admin/tool/behat/cli/init.php --parallel=4
bin/moodle-docker-compose run webserver php admin/tool/behat/cli/run.php

NB: I'm using this Docker VM template, which has docker-compose 1.6.2, which appears to feature run in place of exec.

I was hoping this would launch a 4-threaded behat run. The init step ran fine and installed the 4 databases, but the run step just resulted in this message:

Running single behat site:

  [Behat\Testwork\ServiceContainer\Exception\ConfigurationLoadingException]
  The requested config file does not exist

To re-run failed processes, you can use following command:
php admin/tool/behat/cli/run.php --rerun

I'm guessing there's a need to spin up extra Selenium containers and monkey around with the config.php in order to get this working. It would be lovely if there was a command to do this!

[aspark21]

It's been a while but this does currently already work. Have been routinely using 2 parallel runs for the last 18 months and just completed testing 6-10 parallel runs.

I was thinking the only other thing that could be useful to implement for this was adding support for a timing file so did quite a few tests on our runners (12CPUs 16GBs RAM)

Parallel runs	without	with timing file	Moodle version
2	11h31m	NA	master/4.0dev
2	9h31m	NA	3.9
6	4h16m	NA	master/4.0dev
6	3h27m	NA	3.9
7	3h59m	3h15m	master/4.0dev
8	3h20m	2h57m	master/4.0dev
9	3h44m	NA	master/4.0dev
10	2h59m	2h33m	master/4.0dev
12	2h57m	2h28m	master/4.0dev

(Will take the 10 parallel with timing file into service)

There is some benefit to having the timing file (30mins or so) but I'm not sure it would be that beneficial to have anything specifically setup in moodle-docker for it.

We have a custom config.docker-template.php, mainly in order to define STACK's settings and I've simply added this to it. Relying on #196 to be able to add & retrieve the timing file.

//Behat Timing file
define('BEHAT_FEATURE_TIMING_FILE', '/var/www/behatfaildumps/timing.json');
define('BEHAT_MAX_CMD_LINE_OUTPUT', 500);

There's no big jump in tests failing at any point but there is overall about 20 tests that randomly fail so will need to enable re-runs & the extra time for that is not factored in, in the above.