search

roboconf / roboconf-platform

The core modules and the platform
Apache License 2.0
35 stars 11 forks source link

Parallel execution of unit tests? #806

Closed vincent-zurczak closed 7 years ago

vincent-zurczak commented 7 years ago

We might speed up the build by running some unit tests in parallel. That's the case for the core, the agent and the DM. Those to exclude, for sure, are the plugins.

We should make some benchmarks and verify whether it is worth or not. See http://maven.apache.org/surefire/maven-surefire-plugin/test-mojo.html

vincent-zurczak commented 7 years ago

Test on the core (mvn clean install):

Total time: 28.099 s
[INFO] Finished at: 2017-06-21T21:49:23+02:00
[INFO] Final Memory: 32M/374M

With a configuration to run test classes in parallel (mvn clean install -Dparallel=classes -DthreadCountClasses=4):

[INFO] Total time: 23.854 s
[INFO] Finished at: 2017-06-21T21:56:41+02:00
[INFO] Final Memory: 32M/410M

We use a little more memory, but we spare 4 seconds. Applied to our other modules, it may be worth the modification. To verify on the DM and the agent too.

vincent-zurczak commented 7 years ago
Module Before After
DM 17.528 14.129
Agent 23.950 15.134

It's worth it. We could gain about a minute per build. Just on unit tests. And we can probably gain more with other Maven plug-ins.

vincent-zurczak commented 7 years ago

OK. With modules being built alone, there are few seconds that can be saved. However, there are sometimes race conditions on our tests. As an exemple, building the agent module sometimes fails due to a same file access. In the same way, the Roboconf scheduler can only run alone and it results in an error when two tries to run concurrently (that's probably due to Quartz). Same thing with our REST services (we launch a test server).

I tested both parallel and forkCount strategies, as described on http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html

Either we upgrade our tests (which can be complicated as we have quite a lot), or we make a lot of exceptions (more than what I thought at the beginning), or we consider it will only add more complexity to our build. I choose the last option. But it was interesting to verify this lead.