I recently started needing to repeatedly run a specific test in a gtest framework for the ros2_control/ros2_controllers repository so I implemented gtest_discover_tests in the CMakeLists.txt file so that I could properly pass ctest-args to colcon test and filter according to that test; however, when I did so, I noticed that tests started taking a lot longer to run. For example:
on ctest branch (with gtest_discover_tests):
colcon test --packages-select force_toque_sensor_broadcaster
takes ~3.5 seconds
on master (no gtest discovery):
colcon test --packages-select force_torque_sensor_broadcaster
takes ~0.8 seconds
When digging into the logs of the test's stdout_stderror.log, it looks like the speed difference is coming from colcon executing each test as a separate executing run when gtest_discover_tests is enabled versus running the whole executable once when discovery is not enabled. For example:
When I run the test executable directly, I get the following output:
Running main() from gmock_main.cc
[==========] Running 14 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 14 tests from ForceTorqueSensorBroadcasterTest
[ RUN ] ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet
[ OK ] ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet (15 ms)
[ RUN ] ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet
[ OK ] ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet (12 ms)
[ RUN ] ForceTorqueSensorBroadcasterTest.InterfaceNames_FrameId_NotSet
[ OK ] ForceTorqueSensorBroadcasterTest.InterfaceNames_FrameId_NotSet (10 ms)
[ RUN ] ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_Set
[ OK ] ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_Set (8 ms)
... etc for brevity...
[----------] 14 tests from ForceTorqueSensorBroadcasterTest (162 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 1 test suite ran. (172 ms total)
[ PASSED ] 14 tests.
However when running from colcon test, I get the following output:
UpdateCTestConfiguration from :/mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster/CTestConfiguration.ini
Parse Config file:/mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster/CTestConfiguration.ini
Site: 1f10b170ae5d
Build name: (empty)
Add coverage exclude regular expressions.
Create new tag: 20220419-1251 - Experimental
UpdateCTestConfiguration from :/mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster/CTestConfiguration.ini
Parse Config file:/mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster/CTestConfiguration.ini
Test project /mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
Start 1: ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet
1: Test command: /mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster/test_force_torque_sensor_broadcaster "--gtest_filter=ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet" "--gtest_also_run_disabled_tests"
1: Test timeout computed to be: 10000000
1: Running main() from gmock_main.cc
1: Note: Google Test filter = ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet
1: [==========] Running 1 test from 1 test suite.
1: [----------] Global test environment set-up.
1: [----------] 1 test from ForceTorqueSensorBroadcasterTest
1: [ RUN ] ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet
1: [ERROR] [1650372703.599663675] [test_force_torque_sensor_broadcaster]: 'sensor_name' or at least one 'interface_names.[force|torque].[x|y|z]' parameter has to be specified.
1: [ OK ] ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet (18 ms)
1: [----------] 1 test from ForceTorqueSensorBroadcasterTest (18 ms total)
1:
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test suite ran. (35 ms total)
1: [ PASSED ] 1 test.
1/16 Test #1: ForceTorqueSensorBroadcasterTest.SensorName_InterfaceNames_NotSet ........... Passed 0.15 sec
test 2
Start 2: ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet
2: Test command: /mnt/lund/Code/ros2_control_ws/build/force_torque_sensor_broadcaster/test_force_torque_sensor_broadcaster "--gtest_filter=ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet" "--gtest_also_run_disabled_tests"
2: Test timeout computed to be: 10000000
2: Running main() from gmock_main.cc
2: Note: Google Test filter = ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet
2: [==========] Running 1 test from 1 test suite.
2: [----------] Global test environment set-up.
2: [----------] 1 test from ForceTorqueSensorBroadcasterTest
2: [ RUN ] ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet
2: [ERROR] [1650372703.744065436] [test_force_torque_sensor_broadcaster]: 'frame_id' parameter has to be provided.
2: [ OK ] ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet (14 ms)
2: [----------] 1 test from ForceTorqueSensorBroadcasterTest (14 ms total)
2:
2: [----------] Global test environment tear-down
2: [==========] 1 test from 1 test suite ran. (25 ms total)
2: [ PASSED ] 1 test.
2/16 Test #2: ForceTorqueSensorBroadcasterTest.SensorName_FrameId_NotSet .................. Passed 0.14 sec
test 3
Start 3: ForceTorqueSensorBroadcasterTest.InterfaceNames_FrameId_NotSet
etc for brevity...
which I believe from the lines reading Note: Google Test filter = Force.... on every single test that it's running each test against the executable with the given filter individually when it could run the executable once to run them all.
Does colcon-cmake support a way to batch test runs together when discovery shows that there are tests part of the same executable?
I recently started needing to repeatedly run a specific test in a gtest framework for the
ros2_control/ros2_controllersrepository so I implemented gtest_discover_tests in the CMakeLists.txt file so that I could properly passctest-argsto colcon test and filter according to that test; however, when I did so, I noticed that tests started taking a lot longer to run. For example:colcon test --packages-select force_toque_sensor_broadcastercolcon test --packages-select force_torque_sensor_broadcasterWhen digging into the logs of the test's
stdout_stderror.log, it looks like the speed difference is coming from colcon executing each test as a separate executing run whengtest_discover_testsis enabled versus running the whole executable once when discovery is not enabled. For example:When I run the test executable directly, I get the following output:
However when running from
colcon test, I get the following output:which I believe from the lines reading
Note: Google Test filter = Force....on every single test that it's running each test against the executable with the given filter individually when it could run the executable once to run them all.Does colcon-cmake support a way to batch test runs together when discovery shows that there are tests part of the same executable?