Test discovery not transfering test cases - only in Debug mode

[Bunnyyy189]

runsettings.txt Hi, I got the problem that the test discovery within VS2022 with the adapter is finding test cases with --list-tests, but not transfering them to test explorer... This only happens in Debug mode... Has anybody the same problems or any suggestions?

`C2A-8912: Started Catch2Adapter exe test discovery... C2A-8912: Discover log: ValidateSourceName name: xxx_catch2_uTest Source: D:\WS\dev_debug_01\x64_d\bin\xxx_catch2_uTest.exe Default Verbose Test Discovery: Visual Leak Detector is turned off: disabled by default or config. Settings: D:\WS\dev_debug_01\x64_d\bin\vld.ini. See Wiki for more infos. Set 'VLD_SILENT' enviroment variable to 1 to prevent some outputs of the VLD. Matching test cases: CheckPreOperational-OK D:\WS\dev_debug_01\src\xxx\yy.cpp(203) CheckPreOperational-NOK D:\WS\dev_debug_01\src\xxx\yy.cpp(220) 2 matching test cases

Testcase count: 0 Accumulated Testcase count: 0

C2A-8912: Start adding test cases to discovery sink C2A-8912: Finished adding test cases to discovery sink C2A-8912: Finished Catch2Adapter exe test discovery. No test is available in D:\WS\dev_debug_01\x64_d\bin\xxx_catch2_uTest.exe. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. `

During Release mode, the two test cases are transfered properly, Test count also prints 2, and I can see them in the VS22 Test explorer...

We are using catch2 V3.3.2 and I have the extension for testadapter V2.0.0 installed... Best regards, Bunny

[JohnnyHendriks]

You could try to remove the <DiscoverCommandLine> line from your runsettings-file (i.e., use the default settings --verbosity high --list-tests --reporter xml *). As you are using Catch2 v3.x, it will then automatically use the xml reporter, which is more robust. Due to a format change of the standard list-tests output in Catch2 v3.x, tests with long names can no longer be extracted from the standard output.

[Bunnyyy189]

Unfortunately this brings the same result...

[JohnnyHendriks]

Do you by any chance make use of brackets in the test case names. Test cases with such names are somehow not displayed in Visual Studio Test Explorer. Though, looking closer at the log I do see some unexpected stuff there. Do you add additional logging somehow during discovery? If that is the case that might explain the discovery mechanism getting confused. If you run your test executable on the command line with --verbosity high --list-tests --reporter xml *, is the xml-text generated in the console valid, or are there custom log-lines intertwined with the xml?

[Bunnyyy189]

Hi, thanks for the quick response. No, there are no brackets in the test case names and the xml output is valid. Also it is the same output for Debug and Release executable... (see attachment) xmlLogDebug.txt xmlLogRelease.txt

[JohnnyHendriks]

All looks good there. The only thing I can think of is that another test adapter may be interfering somehow. In case the Google Test and/or the Boost test adapter are enabled, disabling those may solve the problem as well. The weird part in the log for me is

Default Verbose Test Discovery:
Visual Leak Detector is turned off: disabled by default or config. Settings: D:\WS\dev_debug_01\x64_d\bin\vld.ini. See Wiki for more infos. Set 'VLD_SILENT' enviroment variable to 1 to prevent some outputs of the VLD.
Matching test cases:
CheckPreOperational-OK
D:\WS\dev_debug_01\src\xxx\yy.cpp(203)
CheckPreOperational-NOK
D:\WS\dev_debug_01\src\xxx\yy.cpp(220)
2 matching test cases

Testcase count: 0
Accumulated Testcase count: 0

I get the impression that may be logged by something else.

[Bunnyyy189]

I disabled and afterwards uninstalled all other adapters... The log above I got when using Debug in the runsettings file if I understand correctly... I tried looking up in the adapter code, where the cases got lost (maybe in some naming lookup) but couldn't find it.