search

ros2 / launch

Tools for launching multiple processes and for writing tests involving multiple processes.
Apache License 2.0
126 stars 144 forks source link

Feature request: run multiple tests in parallel with `colcon test` #789

Closed LastStarDust closed 4 months ago

LastStarDust commented 4 months ago

I am currently engaged in a ROS2 project aimed at optimizing our testing processes by enabling concurrent execution of tests within a single package using colcon test.

Initially, I attempted to utilize the --parallel-workers flag. However, this approach did not resolve the issue as intended. It facilitates parallel execution across multiple packages, but intra-package test execution remains sequential.

Each test is implemented using the launch_testing framework and integrated into colcon via the add_launch_test macro within the CMakeLists.txt file.

Key specifics include:

I am seeking guidance or practical examples that outline best practices for achieving concurrent test execution at the ROS2 package level.

In case what I am asking turned out to be not supported, I suggest to add it to the list of desired features to be implemented. Thank you in advance!

cottsay commented 4 months ago

I don't have a lot of good news here.

For parallel builds, colcon currently relies on system load tracking in GNU Make to parallelize compilation within each concurrently building CMake package. Though CTest is able to run tests in parallel, it doesn't have the same load tracking parallelization feature that GNU Make does.

To work around this, you could pass --ctest-args -j <NUMBER_OF_WORKERS> to your colcon test invocations, but you should be careful. Many of the tests written for the ROS 2 core packages were not implemented with parallelization in mind and my fail. That doesn't mean you shouldn't write parallelizable tests in your packages, of course.

This story may improve a little bit when we get a GNU job server implemented for colcon-parallel-executor, at which point CTest will be able to borrow job slots from the executor when it want to run parallel tests, but we will probably need to set up a way to "opt-in" packages to parallel testing.

LastStarDust commented 4 months ago

@cottsay Thank you for the quick reply.

--ctest-args -j <NUMBER_OF_WORKERS> works as expected. Also I found out that export CTEST_PARALLEL_LEVEL=<NUMBER_OF_WORKERS> has the same effect.

ros-discourse commented 4 months ago

This issue has been mentioned on ROS Discourse. There might be relevant details there:

https://discourse.ros.org/t/how-to-run-multiple-tests-in-parallel-with-colcon-test/38675/2