Can't run CTest tests in parallel

[dgazzoni]

Brief Issue Summary

Not sure if this is some project misconfiguration on my side, but CTest tests are not run in parallel. For every test <test_name> in my project, I see this in my output tab:

[proc] Executing command: /opt/homebrew/bin/ctest -j10 -C Debug -T test --output-on-failure -R <test_name>

While it actually goes through the motions of running in parallel (by passing -j10), because it filters the list of tests down to a single one (since <test_name> is the name of one single, specific test), there is no opportunity to actually run in parallel.

I get this behavior by running it from the "Testing" pane in the left-side of the VS Code screen: either clicking the button whose tooltip is "Run Tests" and runs all tests, clicking on the "play"-shaped button next to CTest or the same button next to my project folder under CTest. The behavior is also the same if I choose "CMake: Run Tests" or "Test: Run All Tests" from the command palette.

I can't see how this could be the intended behavior, as there's even a setting "Cmake > Ctest: Parallel Jobs" in the extension -- which BTW is set to 0 in my case, and so is "Cmake: Parallel Jobs"; indeed you can see above I have a -j10 in my CTest command line. Could I get some pointers about what I might be doing wrong, perhaps a setting or something in my CMakeLists.txt that is blocking parallel tests? If by chance this is intended behavior, I'd like to turn this into a feature request to get tests to run in parallel as this would speed up my workflow a lot.

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

[benmcmorran]

Thanks for the feedback! Marking this as a feature request for now based on this discussion. For others looking at this issue, please 👍 react to help us decide what to prioritize.

[ebouteillon]

Parallel execution of tests executed by CTest worked with previous version 1.13.45.

It does not work with version 1.14.30. It seems this version runs each test one by one instead of letting CTest schedule the order of execution of tests and use parallel execution when possible. It is visible in logs of #3126

[bilderbuchi]

worked with previous version 1.13.45.

It does not work with version 1.14.30.

It seems to me this should be labelled "regression", then, instead of "feature request"?

[bodiya]

Parallel execution was working for me as well until I updated. I agree with @bilderbuchi that this seems like a regression.

[ben-perry]

This is a pretty huge regression. Before the update I was able to run all my tests (> 300) under a second, now it takes over a minute... Most tests run under 30ms so I am not even sure how running them sequentially is this slow!

[OleksandrKvl]

Was filing a new issue while noticed this one. It's a huge regression in my case, I have ~6.5k mostly simple google tests in 10 binaries, previously it was very fast to run them all. Now it takes around 30 minutes. Just checked, running a single debug binary with all ~650 tests took 2 seconds...

[Coder-256]

Just ran into this as well. It seems that #3032 began running tests individually, one by one, using a for loop:

https://github.com/microsoft/vscode-cmake-tools/blob/787937365e4b96a3144a903ba993b450ebbf2397/src/ctest.ts#L433-L436

For an 8-core system, this is an 8x slowdown. I'm using the command line for now.

[Coder-256]

This also seems related to #3149.

[snehara99]

In the pre-release channel there should be a setting soon called cmake.ctest.allowParallelJobs which will bring back running ctests in parallel when set to true. As a warning though, this might result in the test output results being a bit garbled. Please let me know if this setting works for you all, thanks!

[OleksandrKvl]

@snehara99 I don't see this setting in the current pre-release version, please let us know when it will be there.

[snehara99]

@OleksandrKvl I apologize, I don't think it was released on the marketplace yet at that time. It should be available now though!

[OleksandrKvl]

When I press Run CTest in the Status bar, it doesn't run anything, only opens Testing in the Side bar. OK, in the Side bar, when I press either Run tests (at the top) or Run test (project specific), I don't see any parallelism, it's still one-by-one, my both parallel jobs settings (cmake and ctest) are set to 0. I see that -j12 is passed to individual ctest executions:

Executing command: /opt/homebrew/bin/ctest -j12 -C Debug -T test --output-on-failure -R ^tests_cpp_11_mf\.DynamicArrayRefDeathTest\.Insert2TerminatesIfHoldsNullptr$

but since it runs ctest per each registered test, it has no effect, there's nothing it can parallelize.

[snehara99]

@OleksandrKvl this is after setting cmake.ctest.allowParallelJobs to true correct?

[OleksandrKvl]

@snehara99 yes

[snehara99]

@OleksandrKvl Thanks for confirming! Could you by chance share an extended execution log? I want to see if it's different than what I see after the initial command execution line

[OleksandrKvl]

please clarify what do you mean by "extended execution log"

[snehara99]

exactly what you sent above, but with more lines included

[OleksandrKvl]

OK, here I started and immediately stopped test runner:

[main] Building folder: sbepp 
[build] Starting build
[proc] Executing command: /opt/homebrew/bin/cmake --build /Users/alex/Documents/src/sbepp/build_Debug --config Debug --target all --
[build] ninja: no work to do.
[driver] Build completed: 00:00:00.049
[build] Build finished with exit code 0
[proc] Executing command: /opt/homebrew/bin/ctest -j12 -C Debug -T test --output-on-failure -R ^tests_cpp_11_mf\.CompositeDeathTest\.TerminatesIfAccessedOutOfRange$
[ctest]    Site: alexs-MacBook-Pro.local
[ctest]    Build name: Darwin-g++-13
[ctest] Test project /Users/alex/Documents/src/sbepp/build_Debug
[ctest]     Start 1: tests_cpp_11_mf.CompositeDeathTest.TerminatesIfAccessedOutOfRange
[ctest] 1/1 Test #1: tests_cpp_11_mf.CompositeDeathTest.TerminatesIfAccessedOutOfRange ...   Passed    0.10 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.33 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /opt/homebrew/bin/ctest -j12 -C Debug -T test --output-on-failure -R ^tests_cpp_11_mf\.StaticArrayRefDeathTest\.TerminatesIfAccessedWithoutEnoughStorage$
[ctest]    Site: alexs-MacBook-Pro.local
[ctest]    Build name: Darwin-g++-13
[ctest] Test project /Users/alex/Documents/src/sbepp/build_Debug
[ctest]     Start 2: tests_cpp_11_mf.StaticArrayRefDeathTest.TerminatesIfAccessedWithoutEnoughStorage
[ctest] 1/1 Test #2: tests_cpp_11_mf.StaticArrayRefDeathTest.TerminatesIfAccessedWithoutEnoughStorage ...   Passed    0.11 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.34 sec
[ctest] CTest finished with return code 0
[proc] Executing command: /opt/homebrew/bin/ctest -j12 -C Debug -T test --output-on-failure -R ^tests_cpp_11_mf\.StaticArrayRefDeathTest\.SubscriptTerminatesIfAccessedOutOfRange$
[ctest]    Site: alexs-MacBook-Pro.local
[ctest]    Build name: Darwin-g++-13
[ctest] Test project /Users/alex/Documents/src/sbepp/build_Debug
[ctest]     Start 3: tests_cpp_11_mf.StaticArrayRefDeathTest.SubscriptTerminatesIfAccessedOutOfRange
[ctest] 1/1 Test #3: tests_cpp_11_mf.StaticArrayRefDeathTest.SubscriptTerminatesIfAccessedOutOfRange ...   Passed    0.03 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   0.25 sec
[ctest] CTest finished with return code 0

[snehara99]

Thank you, I am able to repro the issue and I will have a fix out soon. In the meantime, invoking tests from the Project Status View should work in parallel.

[snehara99]

This is related to issue #3151 so I'm making a note here

[rjra100]

I for one am finding the Project Status View's interface a bit unintuitive: the "[All tests]" entry under "Tests" doesn't actually have any run options, and the "Play" button at the top appears to be equivalent to launching the selected target under the "launch" section. So it's very easy to click "All tests" and hit "play" to run them, resulting in it running something else entirely. Possibly a "Tests" button at the top would help.

Personally, I don't tend to use Project Status View anyway - I want to run my tests off the Testing pane. If there's a one-click option to run all the tests via the CMake pane or the status bar, that's nice, but that's not primarily where I expect to be running the tests from - and I want them to work in parallel there too. Pretty please? :)

[snehara99]

@rjra100 Thank you for your feedback! We will take it into account as we are currently redesigning the Project Status View. It would be helpful if you could make a new issue with the request of running tests in parallel from the Test Explorer as it is a bit out of scope for this issue. We will take a look at it as soon as we can :)

[snehara99]

@OleksandrKvl If you upgrade to the newest pre-release channel, the Run CTests option should also work to run your tests in parallel if you have cmake.ctest.allowParallelJobs set to true. It will now bypass the Test Explorer entirely (which you can open from the Activity Bar on the left if needed). Please let me know if this works for you, thanks!

[dgazzoni]

@snehara99 thanks for the work done until now. I don't mean to sound ungrateful, but as I reported when I opened the issue, I usually run tests from the "Testing" pane, using the button "Run tests" with the double-play-like icon. Although running the tests by clicking on "Run CTest" in the status bar is now parallelized, running from the "Testing" pane isn't, and is still slow.

Earlier you said this is a bit out of scope for this issue, but I'd like to disagree. It's exactly the original use case reported, and as such the present solution fixes a related issue, but not the exact same issue. I'm not sure how hard it would be to adapt this to work in the "Testing" pane. If you feel it would be better to open another issue specifically targeting this way of running the tests, let me know and I'll do it.

[OleksandrKvl]

@snehara99 thanks, now it works as you explained. In my case, 6.5k tests were executed in 37 seconds and I still wonder why it's so slow comparing to running a test binary by hand which takes only 2 seconds (the whole suite has 10 binaries). But I guess it's related to ctest itself, not to the extension.

[OleksandrKvl]

Executing the same ctest -j12 command from VSCode terminal takes only 28 seconds.

[OleksandrKvl]

FYI, since ctest runs each test in separate process, when you use gtest_discover_tests, ctest will run each of those discovered unit tests in a separate process, hence the slowdown comparing to running all of them in a single process at once. I tried to replace gtest_discover_tests with a plain add_test, now ctest sees it a single test but if you have multiple test binaries (like me), total run time is still not optimal. In my case, running ctest -j12 with a single binary takes ~2 seconds:

[proc] Executing command: /opt/homebrew/bin/ctest -j12 -C Debug -T test --output-on-failure -R ^tests_cpp_11_mf$
[ctest]    Site: alexs-MacBook-Pro.local
[ctest]    Build name: Darwin-g++-13
[ctest] Test project /Users/alex/Documents/src/sbepp/build_Debug
[ctest]     Start 1: tests_cpp_11_mf
[ctest] 1/1 Test #1: tests_cpp_11_mf ..................   Passed    1.98 sec
[ctest] 
[ctest] 100% tests passed, 0 tests failed out of 1
[ctest] 
[ctest] Total Test time (real) =   2.02 sec
[ctest] CTest finished with return code 0

but running ctest -j12 with 10 binaries takes ~16 seconds:

/opt/homebrew/bin/ctest -j12 -C Debug --output-on-failure
Test project /Users/alex/Documents/src/sbepp/build_Debug/test
      Start  1: tests_cpp_11_mf
      Start  2: tests_cpp_11_tf
      Start  3: tests_cpp_14_mf
      Start  4: tests_cpp_14_tf
      Start  5: tests_cpp_17_mf
      Start  6: tests_cpp_17_tf
      Start  7: tests_cpp_20_mf
      Start  8: tests_cpp_20_tf
      Start  9: tests_cpp_23_mf
      Start 10: tests_cpp_23_tf
 1/10 Test  #2: tests_cpp_11_tf ..................   Passed   16.57 sec
 2/10 Test  #3: tests_cpp_14_mf ..................   Passed   16.59 sec
 3/10 Test  #5: tests_cpp_17_mf ..................   Passed   16.60 sec
 4/10 Test  #4: tests_cpp_14_tf ..................   Passed   16.61 sec
 5/10 Test  #1: tests_cpp_11_mf ..................   Passed   16.63 sec
 6/10 Test  #6: tests_cpp_17_tf ..................   Passed   16.63 sec
 7/10 Test  #8: tests_cpp_20_tf ..................   Passed   16.64 sec
 8/10 Test  #7: tests_cpp_20_mf ..................   Passed   16.66 sec
 9/10 Test  #9: tests_cpp_23_mf ..................   Passed   16.67 sec
10/10 Test #10: tests_cpp_23_tf ..................   Passed   16.67 sec

100% tests passed, 0 tests failed out of 10

Total Test time (real) =  16.71 sec

This seems to be a known problem in CTest and might be fixed in future releases.

[snehara99]

@OleksandrKvl Thanks for investigating the root of the issue. If a fix is released for CTest in the future, then we'll integrate it asap. If you like, you can create a new issue to track this problem moving forward. As for the original issue, the fix should now be available in the main release channel, so I am going to close this issue as completed.

[Zitrax]

I am using CMake Tools 1.16.2 (Pre-release), and I use gtest_discover_tests and enabled cmake.ctest.allowParallelJobs. But I can't see the tests running in parallel. I see no -j arg passed to ctest in the output pane.

Is it supposed to work with this version?

[dgazzoni]

@snehara99 can you please advise with regards to my previous comment?

As the original submitter, this doesn't fix the issue at all for me, due to the different path I take for running the tests (and which was reported as such when I opened the issue). Should I open a new issue

[snehara99]

@snehara99 can you please advise with regards to my previous comment?

As the original submitter, this doesn't fix the issue at all for me, due to the different path I take for running the tests (and which was reported as such when I opened the issue). Should I open a new issue

Hi @dgazzoni, sorry for the late reply. I must have missed your comment while replying to others. I agree that your issue is still in scope with regards to the original post, however it would be very helpful for tracking on our end if you could create a new issue with this specific request to run parallel tests from the testing pane. Thank you for bumping this, we will start investigating as soon as the new issue is created!

[snehara99]

I am using CMake Tools 1.16.2 (Pre-release), and I use gtest_discover_tests and enabled cmake.ctest.allowParallelJobs. But I can't see the tests running in parallel. I see no -j arg passed to ctest in the output pane.

Is it supposed to work with this version?

Hi @Zitrax could you please start a new issue so we can track this separately? I think this may be a slightly different issue than the one that was addressed here, and we will need to do more investigation. Thanks!

[dgazzoni]

@snehara99 can you please advise with regards to my previous comment? As the original submitter, this doesn't fix the issue at all for me, due to the different path I take for running the tests (and which was reported as such when I opened the issue). Should I open a new issue

Hi @dgazzoni, sorry for the late reply. I must have missed your comment while replying to others. I agree that your issue is still in scope with regards to the original post, however it would be very helpful for tracking on our end if you could create a new issue with this specific request to run parallel tests from the testing pane. Thank you for bumping this, we will start investigating as soon as the new issue is created!

Thanks. I just opened #3322 to cover this use case.