CTest integration does not work with Catch2's `catch_discover_tests()`

samwarring commented 1 year ago

Brief Issue Summary

When I use Catch2's catch_discover_tests command, those tests don't always get populated in the test explorer, and consequently running the test cases via the CTest integration does not work.

Steps to reproduce:

I have a small repo that repoduces the problem. It uses CMakePresets, and obtains Catch2 with vcpkg (and vcpkg is included as a submodule). The CMakePresets in this revision of the repo only work on Linux.
Open the repo in VSCode
Select any configure preset (e.g. "GCC Debug")
Select the build preset ("gcc-debug")
Select the test preset ("gcc-debug")
Build the project
Run CTest

Expected result

The test cases are executed, and can be browsed in the test explorer.

Actual result

The output window shows "No tests were found!!!" There is one test case in the test explorer named project_test_NOT_BUILT-b12d07c.

Analysis

The NOT_BUILT test case is added by Catch2's catch_discover_tests. If you inspect the build directory, you can find that CTestTestfile.cmake eventually executes:

if(EXISTS "path/to/project_tests-b12d07c.cmake")
  include("path/to/project_tests-b12d07c.cmake")
else()
  add_test(project_test_NOT_BUILT-b12d07c project_test_NOT_BUILT-b12d07c)
endif()

The path it's looking for does exist, but I'm guessing that vscode-cmake-tools processes this file too early, and never re-processes it after the tests are actually built.

CMake Tools Diagnostics

{
  "os": "linux",
  "vscodeVersion": "1.78.2",
  "cmtVersion": "1.14.31",
  "configurations": [
    {
      "folder": "/home/sam/git/github.com/samwarring/cpp_project_template",
      "cmakeVersion": "3.25.2",
      "configured": true,
      "generator": "Unix Makefiles",
      "usesPresets": true,
      "compilers": {
        "C": "/usr/bin/gcc",
        "CXX": "/usr/bin/g++"
      }
    }
  ],
  "cpptoolsIntegration": {
    "isReady": true,
    "hasCodeModel": true,
    "activeBuildType": "Debug",
    "buildTypesSeen": [
      "Debug"
    ],
    "requests": [],
    "responses": [],
    "partialMatches": [],
    "targetCount": 2,
    "executablesCount": 1,
    "librariesCount": 1,
    "targets": [
      {
        "name": "project_lib",
        "type": "STATIC_LIBRARY"
      },
      {
        "name": "project_test",
        "type": "EXECUTABLE"
      }
    ]
  },
  "settings": [
    {
      "communicationMode": "automatic",
      "useCMakePresets": "auto",
      "configureOnOpen": false
    }
  ]
}

Debug Log

[main] Building folder: cpp_project_template 
[main] Saving open files before configure/build
[build] Starting build
[driver] Start build 
[proc] Executing command: /usr/bin/cmake --build /home/sam/git/github.com/samwarring/cpp_project_template/build/gcc-release
[proc]   with environment: {"USER":"sam","VSCODE_WSL_EXT_LOCATION":"/mnt/c/Users/samwa/.vscode/extensions/ms-vscode-remote.remote-wsl-0.78.8","SHLVL":"1","WT_PROFILE_ID":"{2c4de342-38b7-51cf-b940-2309a097f518}","HOME":"/home/sam","WSL_DISTRO_NAME":"Ubuntu","WAYLAND_DISPLAY":"wayland-0","LOGNAME":"sam","NAME":"Code","WSL_INTEROP":"/run/WSL/8_interop","PULSE_SERVER":"/mnt/wslg/PulseServer","_":"/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code/bin/code","TERM":"xterm-256color","PATH":"/home/sam/.vscode-server/bin/b3e4e68a0bc097f0ae7907b217c1119af9e03435/bin/remote-cli:/home/sam/.local/bin:/home/sam/bin:/home/sam/.gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/WindowsApps/Microsoft.WindowsTerminal_1.16.10262.0_x64__8wekyb3d8bbwe:/mnt/c/Python311/Scripts/:/mnt/c/Python311/:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/PuTTY/:/mnt/c/gstreamer/1.0/msvc_x86_64/bin:/mnt/c/Program Files/Meson/:/mnt/c/Program Files/Dependencies_x64_Release:/mnt/c/Program Files/Amazon/AWSCLIV2/:/mnt/c/Program Files/nodejs/:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/CMake/bin:/mnt/c/Program Files/Clang Format AutoVersion/bin:/mnt/c/Users/samwa/.cargo/bin:/mnt/c/Users/samwa/AppData/Local/Programs/Python/Python310/Scripts/:/mnt/c/Users/samwa/AppData/Local/Programs/Python/Python310/:/mnt/c/Users/samwa/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/samwa/AppData/Roaming/npm:/mnt/c/Users/samwa/bin:/snap/bin","WT_SESSION":"a6e4e21f-ae95-4d98-8e8a-c36d14bd9c7a","XDG_RUNTIME_DIR":"/mnt/wslg/runtime-dir","DISPLAY":":0","LANG":"C.UTF-8","SHELL":"/bin/bash","PWD":"/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code","HOSTTYPE":"x86_64","WSLENV":"ELECTRON_RUN_AS_NODE/w:WT_SESSION::WT_PROFILE_ID\n","VSCODE_HANDLES_SIGPIPE":"true","GENICAM_GENTL64_PATH":"/opt/spinnaker/lib/flir-gentl","MOTD_SHOWN":"update-motd","LS_COLORS":"rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:","GEM_HOME":"/home/sam/.gems","LESSCLOSE":"/usr/bin/lesspipe %s %s","LESSOPEN":"| /usr/bin/lesspipe %s","FLIR_GENTL64_CTI":"/opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti","XDG_DATA_DIRS":"/usr/local/share:/usr/share:/var/lib/snapd/desktop","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"en\",\"availableLanguages\":{}}","VSCODE_CWD":"/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code","ELECTRON_RUN_AS_NODE":"1","VSCODE_IPC_HOOK_CLI":"/mnt/wslg/runtime-dir/vscode-ipc-f621a172-2151-4d51-b040-f6d93d4fae13.sock","APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL":"1","VSCODE_L10N_BUNDLE_LOCATION":""}
[build] [ 25%] Building CXX object src/project_lib/CMakeFiles/project_lib.dir/math.cpp.o
[build] [ 50%] Linking CXX static library libproject_lib.a
[build] [ 50%] Built target project_lib
[build] [ 75%] Building CXX object src/project_test/CMakeFiles/project_test.dir/math_test.cpp.o
[build] [100%] Linking CXX executable project_test
[build] [100%] Built target project_test
[driver] Build completed: 00:00:01.120
[cmakefileapi-parser] Read reply folder: /home/sam/git/github.com/samwarring/cpp_project_template/build/gcc-release/.cmake/api/v1/reply
[cmakefileapi-parser] Found index files: ["cache-v2-ebc5fcaa585c3f6482e8.json","cmakeFiles-v1-5a435cc3bbf1aa9ba1b6.json","codemodel-v2-11a6390a43579dc6d64c.json","directory-.-Release-f5ebdc15457944623624.json","directory-src-Release-b6cf2213fc5053b67c22.json","directory-src.project_lib-Release-20cd0408c321113af7df.json","directory-src.project_test-Release-9b146bb7e0051890da34.json","index-2023-05-29T17-24-39-0781.json","target-project_lib-Release-4b81fddfee67baa71209.json","target-project_test-Release-3cb389dd6f38f94c3ca1.json","toolchains-v1-2781238ec0fa0b9996f9.json"]
[cache] Reading CMake cache file /home/sam/git/github.com/samwarring/cpp_project_template/build/gcc-release/CMakeCache.txt
[cache] Parsing CMake cache string
[build] Build finished with exit code 0
[extension] [6094] cmake.build finished (returned 0)

[proc] Executing command: /usr/bin/ctest -T test --output-on-failure -R ^project_test_NOT_BUILT-b12d07c$
[proc]   with environment: {"USER":"sam","VSCODE_WSL_EXT_LOCATION":"/mnt/c/Users/samwa/.vscode/extensions/ms-vscode-remote.remote-wsl-0.78.8","SHLVL":"1","WT_PROFILE_ID":"{2c4de342-38b7-51cf-b940-2309a097f518}","HOME":"/home/sam","WSL_DISTRO_NAME":"Ubuntu","WAYLAND_DISPLAY":"wayland-0","LOGNAME":"sam","NAME":"Code","WSL_INTEROP":"/run/WSL/8_interop","PULSE_SERVER":"/mnt/wslg/PulseServer","_":"/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code/bin/code","TERM":"xterm-256color","PATH":"/home/sam/.vscode-server/bin/b3e4e68a0bc097f0ae7907b217c1119af9e03435/bin/remote-cli:/home/sam/.local/bin:/home/sam/bin:/home/sam/.gems/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/WindowsApps/Microsoft.WindowsTerminal_1.16.10262.0_x64__8wekyb3d8bbwe:/mnt/c/Python311/Scripts/:/mnt/c/Python311/:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0/:/mnt/c/WINDOWS/System32/OpenSSH/:/mnt/c/Program Files/dotnet/:/mnt/c/Program Files/Git/cmd:/mnt/c/Program Files/PuTTY/:/mnt/c/gstreamer/1.0/msvc_x86_64/bin:/mnt/c/Program Files/Meson/:/mnt/c/Program Files/Dependencies_x64_Release:/mnt/c/Program Files/Amazon/AWSCLIV2/:/mnt/c/Program Files/nodejs/:/mnt/c/ProgramData/chocolatey/bin:/mnt/c/Program Files/CMake/bin:/mnt/c/Program Files/Clang Format AutoVersion/bin:/mnt/c/Users/samwa/.cargo/bin:/mnt/c/Users/samwa/AppData/Local/Programs/Python/Python310/Scripts/:/mnt/c/Users/samwa/AppData/Local/Programs/Python/Python310/:/mnt/c/Users/samwa/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/samwa/AppData/Roaming/npm:/mnt/c/Users/samwa/bin:/snap/bin","WT_SESSION":"a6e4e21f-ae95-4d98-8e8a-c36d14bd9c7a","XDG_RUNTIME_DIR":"/mnt/wslg/runtime-dir","DISPLAY":":0","LANG":"C.UTF-8","SHELL":"/bin/bash","PWD":"/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code","HOSTTYPE":"x86_64","WSLENV":"ELECTRON_RUN_AS_NODE/w:WT_SESSION::WT_PROFILE_ID\n","VSCODE_HANDLES_SIGPIPE":"true","GENICAM_GENTL64_PATH":"/opt/spinnaker/lib/flir-gentl","MOTD_SHOWN":"update-motd","LS_COLORS":"rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:","GEM_HOME":"/home/sam/.gems","LESSCLOSE":"/usr/bin/lesspipe %s %s","LESSOPEN":"| /usr/bin/lesspipe %s","FLIR_GENTL64_CTI":"/opt/spinnaker/lib/flir-gentl/FLIR_GenTL.cti","XDG_DATA_DIRS":"/usr/local/share:/usr/share:/var/lib/snapd/desktop","VSCODE_AMD_ENTRYPOINT":"vs/workbench/api/node/extensionHostProcess","VSCODE_HANDLES_UNCAUGHT_ERRORS":"true","VSCODE_NLS_CONFIG":"{\"locale\":\"en\",\"osLocale\":\"en\",\"availableLanguages\":{}}","VSCODE_CWD":"/mnt/c/Users/samwa/AppData/Local/Programs/Microsoft VS Code","ELECTRON_RUN_AS_NODE":"1","VSCODE_IPC_HOOK_CLI":"/mnt/wslg/runtime-dir/vscode-ipc-f621a172-2151-4d51-b040-f6d93d4fae13.sock","APPLICATION_INSIGHTS_NO_DIAGNOSTIC_CHANNEL":"1","VSCODE_L10N_BUNDLE_LOCATION":""}
[ctest] Cannot find file: /home/sam/git/github.com/samwarring/cpp_project_template/build/gcc-release/DartConfiguration.tcl
[ctest]    Site: 
[ctest]    Build name: (empty)
[ctest] Create new tag: 20230529-1725 - Experimental
[ctest] Test project /home/sam/git/github.com/samwarring/cpp_project_template/build/gcc-release
[ctest] Cannot find file: /home/sam/git/github.com/samwarring/cpp_project_template/build/gcc-release/DartConfiguration.tcl
[ctest] No tests were found!!!
[ctest] CTest finished with return code 0
[ctest] [object Object]
[extension] [5540] cmake.ctest finished (returned 0)

Additional Information

No response

samwarring commented 1 year ago

I also tried a similar setup using GoogleTest and the gtest_discover_tests command. I see the same behavior were no tests are found.

I also tried with the gtest_add_tests command, and this did work as expected. This command computes the list of tests at configure-time by parsing the test case source files - whereas gtest_discover_tests and catch_discover_tests invoke the built test-binary to get the list of tests, and therefore cannot populate the full list until build-time.

samwarring commented 1 year ago

I've found the easiest workaround is to simply refresh the tests in the test explorer, and that re-populates the list. However, I think this is less than ideal because that never happens automatically. To me, an easy fix would be to automatically refresh the tests after every build.

samwarring commented 1 year ago

Also, I noticed other recent issues were tagged with milestones, but not this one. The only ones without them are not triaged, or "more info needed". I hope there is a plan to address this in the future. Thanks!

stephenlevitt commented 1 year ago

I can confirm that the same issue is present when using doctest_discover_tests.

xisui-MSFT commented 1 year ago

The behavior looks inconsistent. Feels like a VS Code issue. Trying to make a simplified repro...

xisui-MSFT commented 1 year ago

@samwarring @stephenlevitt Could you test latest VS Code insiders (1.82.0) and check if this issue still repros?

samwarring commented 1 year ago

I tried with VS Code insiders v1.82.0, and it looks like the issue is fixed. Thank you!

Now, when I choose a new configure preset, the test explorer shows "Select a test preset to discover tests." Indeed, doing so populates the test explorer. 👍

One minor issue with the latest behavior - when I add a new test to my source files, I still need to press "Refresh Tests" in order for it to appear in the test explorer. Not really a bug, but it would be great if were added automatically when I "Run all tests".

Another corner case - if I delete the build directory, I can reconfigure and rebuild everything by pressing the "Build" button in the status bar, but this will not repopulate the test explorer. Even though the project is already configured at this point, the test explorer does not update until I explicitly "CMake: Configure" the project again. I can live with this.

Thanks again!

xisui-MSFT commented 1 year ago

when I add a new test to my source files, I still need to press "Refresh Tests" in order for it to appear in the test explorer.

If you find this a concern, please create a new ticket.

if I delete the build directory, I can reconfigure and rebuild everything by pressing the "Build" button in the status bar, but this will not repopulate the test explorer

I wasn't able to repro this, either using *_discover_tests() or add_test(). Can you provide more detailed steps?

samwarring commented 1 year ago

I wasn't able to repro this, either using *_discover_tests() or add_test(). Can you provide more detailed steps?

I tried this again, and I couldn't repo either. It seems I was mistaken. Thanks for your help. I consider this issue resolved.

microsoft / vscode-cmake-tools