Support parallel execution of test steps/cases/modules inside an ETS

[michellutz]

Background and Motivation:

Performance optimisation is required to reduce startup time and validation time; reduction of startup time will simplify cloud deployment horizontal scaling, while reduction in validation time will be helpful while integrating ETF with INSPIRE Geoportal, or any other Metadata related workflow/pipeline.

One identified performance issue is that test inside each Executable Test Suite are executed sequentially, not in parallel.

Proposed change

Include parallelisation inside the ETS, at the level of Test Modules, Test Cases, Test Steps or Assertions. The responsibility of the threads remains in the ETS itself, i.e. it is up to the test developer and the capabilities of the test engine whether and how items are parallelised.

There must be a communication between the ETS and the Framework:

• the ETS must inform ETF if it supports parallel execution with other ETSs.
• ETF then has to run a limited number of ETS in parallel. The number of parallel executed ETS depends on the total system resources and the current utilisation by a number of users.
• The ETS must indicate whether it can execute test items (Test Cases, Assertions, etc) in parallel.
• The ETF then has to decide how many threads are allowed to run concurrently based on system load and inform the ETS.

Alternatives

n/a

Funding

JRC will be ready to fund within its current development contract.

Additional information

n/a

[michellutz]

Split off from #14 as agreed in the 4th SG meeting on 2018-09-04.

[jenriquesoriano]

To do: analyze ETS structure to assess if the tests can be run in parallel and run a test to see the difference

[carlospzurita]

For the SoapUI tests, we found on the STestCaseRunner class that the TestCases on an TestSuite can call each other. So, in each task, all TestCases that are independent can be run in parallel. For TestSteps, SoapUI handles them and we cannot change the behaviour.

[jonherrmann]

Possible blocker for BaseX ETS: https://github.com/BaseXdb issue 1329

[carlospzurita]

Thanks for the reference @jonherrmann . It seems that this issue is not resolved yet. Is there any BaseX ETS using this feature?

[jonherrmann]

Nope. Just found the issue when I was checking something the BaseX repo...