Tests won´t show up in VS 2019 Test Explorer

[richSpiegel]

Hi there,

I was wondering why my google tests are not displayed within the Test Explorer window. When I initially created a test project, a simple sample test will show up in the test explorer, however when starting to include necessary libs and source files it seems to effect the tests disappear (just an assumption, that this is the reason)! The tests run fine in the debugger! So I am about to work through the troubleshoot and stopped at the part about the --getest_list_tests. When I add this parameter, I will get a list of all tests but the tests won´t run. I read in the reference of gtest:

None of the tests listed are actually run if the flag is provided. There is no corresponding environment variable for this flag.

So did they change the behaviour ever since the version csoltenborn refers to? If so, I would proceed with the trouble shoot and accept that the tests won´t run with this parameter provided.

Any idea what else could be the problem with no test displayed in test explorer? Btw. I am using gtest framework Release 1.11.0

[csoltenborn]

The VS test framework works in 2 phases: Discovery (=> tests are displayed in the test explorer, but not run) and execution. For the former, GTA uses the gtest_list_tests flag, so this is perfectly normal.

To figure out why your tests are not discovered any more, first follow the trouble shooting guide. My initial guess would be that your change prevents gtest from properly listing the tests - are they displayed on the console if you run your executable with the option?

If this doesn't help, please activate all debugging options in the settings, rebuild your project (which triggers test discovery), and attach the resulting output from the Test Output window as a text file - maybe we (i.e., I ;-) can see the reason this way...

[csoltenborn]

Closed due to inactivity - feel free to reopen...

[richSpiegel]

Sorry, was gone for a break! You say:

• are they displayed on the console if you run your executable with the option?

What option do you mean, gtest_list_tests? Yes, they all (TEST_P - each param test) will be dispayed in console but won´t be executed which seems normal according to your statement.

I link some libraries like this: $(SolutionDir)$(Configuration);$(SolutionDir)lib; in the project properties "Additional Library Directories", could that be a problem? You mention something in the troubleshoot concerning the "PATH extension" which I don´t quite understand...

[csoltenborn]

GTA does not make use of any VS settings for technical reasons. If these libraries are required for test execution, then I wonder why your executable can be run from the console... Anyways: the PATH extension option is for this very purpose - use it to add the required libraries to the path.

Once more: if in the console, you cd into the directory the executable is located at, and if you run the very same command that GTA prints out in the Test output window, your tests should be discovered (because that's what GTA does). If this is not the case, fix it :-) - if they do run, I need the complete output.

[richSpiegel]

Ok, that is good to know! I assumed GTA uses VS settings as well. The problem is most likely the working directory, which I redirected to a directory which contains several dlls needed for execution. Running it from console in its original directory fails! So, would I add the particular working directory to the path extension in the GTA settings?

[csoltenborn]

You would first RTFM :-)

It kind of sounds like the Working directory option might be useful here, probably in combination with placeholders. The other possibility would be to add each dll with the *Path extension" option.

[richSpiegel]

Alright, thanks for those hints ;-) It is still not working but I will play around with both options and see what happends. What I just don´t understand is how the GTA settings would have influence on the test execution via console? Is a rebuild always necessary after changing settings?

[csoltenborn]

They don't - what made you think that? It's just that you can check on the console whether it should work with GTA, and under what circumstances (e.g., using a particular working directory). And then you can configure GTA such that it invokes the executable under the same circumstances.

[csoltenborn]

Here's how I would tackle the problem (assuming that you can successfully list your tests by cd-ing into directory C:\\foo (which does not contain your executable bar.exe) and just running <some path>\bar.exe --gtest_list_tests):

• configure Working dir in VS options:
• check if discovery works by rebuilding project (tests should appear in Test Explorer)
• switch the definition of Working dir to something relative to e.g. your test executable, making use of the available placeholders (e.g., $(ExecutableDir)\..\foo)
• check if discovery still works
• add a solution settings file (or a project settings file) to your solution with that setting (see example) to make sure that tests are discovered if somebody freshly checks out the solution
• remove above setting from VS options
• check if discovery still works
• [optional] donate to the GTA project ;-)

[richSpiegel]

When "...cd-ing into directory" is supposed to be my working directory, then yes, gtest_list_tests works as well as executing it without this option! However, setting the working dir and rebuilding wouldn´t reveal the tests. Btw. I am using version 0.18.0.1432, where my settings mask looks different to yours - is that a problem?

Could there be a problem with the "Output Directory" in VS? I am using a solution wide setting: $(SolutionDir)$(Configuration)\

[csoltenborn]

First of all, I see a different problem: You have installed Test Adapter for Google Test (the MS clone of my adapter which comes with VS) as well as Google Test Adapter (my adapter), which does not work - please uninstall one of them (and if that happens to be my adapter, please get support from the MS guys).

It's hard to say what exactly happens without the log output from the adapter. Please enable all debugging options, rebuild, and attach the output of the Test Output window as a file.

Any, by the way, I asked for this in my very first reply. I do not want to come across unfriendly, but this is really starting to get annoying. Please do not expect any more answers from me before you have read the docs, and provided all information I need.

[richSpiegel]

Sorry for the odyssey, but as long as I configure in the wrong Test Adapter your docs won´t be effective there - or did I overlooked the passage with those two adapters and one to be uninstalled?! Anyways, it was my fault and I thank you for your support (I will forward a donation query to my boss!).

[csoltenborn]

Did this indeed fix the issue? Sweet - glad that we got it working finally... (and thanks for the donation request!).

Btw (and I might loose my donation now ;-) ): The option Prefix output with [GTA] is rather useful for similar cases - we could have solved this within two steps :-)

I agree that this info should be contained in the docs - the reason that it isn't is that having both adapters installed resulted in errors. They have apparently been fixed by MS in the meantime, making this problem much more subtle... I will add according information soon.

[richSpiegel]

Ya, that fixed it, however now I have the problem passing the working directory via the settings file. Could you have a quick look, what I do wrong? The file is called "aqs.gta.runsettings" and is located in the solution folder of aqs.sln. `<RunSettings

c:\mg\aqs_test\medigraf .*Tests.*_gta.exe c:\mg\aqs_test\medigraf .*Tests.*_gta.exe c:\mg\aqs_test\medigraf

</RunSettings>`

The test output is:

Executing all tests in project: aqs_SoftDapTests ========== Starting test run ========== [GTA] Visual Studio Version: VS2019 [GTA] Google Test Adapter: Test execution starting... [GTA] Solution settings: AdditionalPdbs: '', AdditionalTestExecutionParam: '', BatchForTestSetup: '', BatchForTestTeardown: '', BreakOnFailure: False, CatchExceptions: True, DebuggerKind: Native, EnvironmentVariables: '', EnvironmentVariablesParser: GoogleTestAdapter.Helpers.EnvironmentVariablesParser, ExitCodeTestCase: '', HelperFilesCache: GoogleTestAdapter.Settings.HelperFilesCache, KillProcessesOnCancel: False, MaxNrOfThreads: 16, MissingTestsReportMode: ReportAsNotFound, NrOfTestRepetitions: 1, OutputMode: Debug, ParallelTestExecution: False, ParseSymbolInformation: True, PathExtension: '', PrefixOutputWithGta: True, PrintTestOutput: True, RunDisabledTests: False, SeverityMode: Automatic, ShuffleTests: False, ShuffleTestsSeed: 0, SkipOriginCheck: False, SummaryMode: WarningOrError, TestDiscoveryRegex: '', TestDiscoveryTimeoutInSeconds: 30, TestNameSeparator: '', TimestampMode: Automatic, TraitsRegexesAfter: {}, TraitsRegexesBefore: {}, WorkingDir: '$(ExecutableDir)' [GTA] No test case filter provided [GTA] Running 162 tests... [GTA] No settings configured for test executable 'C:\mg\aqs\Debug\aqs_SoftDapTests.exe'; running with solution settings [GTA] >>>>>>>>>>>>>>> Output of command 'C:\mg\aqs\Debug\aqs_SoftDapTests.exe --gtest_output="xml:C:\Users\examion\AppData\Local\Temp\tmpE805.tmp" --gtest_catch_exceptions=1 --gtest_break_on_failure=0' [GTA] <<<<<<<<<<<<<<< End of Output [GTA] Executable C:\mg\aqs\Debug\aqs_SoftDapTests.exe returned with exit code -1073741515 [GTA] Reported 0 test results to VS during test execution, executable: 'C:\mg\aqs\Debug\aqs_SoftDapTests.exe' [GTA] Test result file C:\Users\examion\AppData\Local\Temp\tmpE805.tmp could not be parsed (completely) - your test executable has probably crashed. Exception message: Das Stammelement ist nicht vorhanden. [GTA] 162 test cases seem to not have been run - possible causes:

• A test run is repeated, but tests have changed in the meantime

• A test dependency has been removed or changed without Visual Studio noticing Failed to add result for test 'ValidateCanonDap/aqsSoftDapCanonTestSuite.ValidateCanonDap/0' with ID '03241a1d-5ed6-cf0b-4c75-050e9a7bc853'. [GTA] Reported 162 test results to VS, executable: 'C:\mg\aqs\Debug\aqs_SoftDapTests.exe', duration: 00:00:00.0601451 [GTA] Execution took 00:00:00.5608551 [GTA] No settings configured for test executable 'C:\mg\aqs\Debug\aqs_SoftDapTests.exe'; running with solution settings [GTA] Google Test execution completed, overall duration: 00:00:00.6025033. [GTA]

  The following warnings and errors occured during test execution: [GTA] Test result file C:\Users\examion\AppData\Local\Temp\tmpE805.tmp could not be parsed (completely) - your test executable has probably crashed. Exception message: Das Stammelement ist nicht vorhanden. [GTA] 162 test cases seem to not have been run - possible causes:

• A test run is repeated, but tests have changed in the meantime
• A test dependency has been removed or changed without Visual Studio noticing ========== Test run finished: 162 Tests (0 Passed, 0 Failed, 0 Skipped) run in 633,8 ms ==========>

[csoltenborn]

I see a broken XML tag at the very beginning (<RunSettings) - could that be the reason? Otherwise, I have no idea - I do not recall any problems in finding the solution settings...

[richSpiegel]

Ah, sorry I did only copy a snippet of the original XML Runsettings where I forgot to close the tag. I reference your xml schema at the beginning - this is the actual content which does not work: `<?xml version="1.0" encoding="utf-8"?> <RunSettings xmlns="GoogleTestAdapterSettings.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.examion.com/ ../UnitTestUtils/GoogleTestSettings/GoogleTestAdapterSettings.xsd">

c:\mg\aqs_test\medigraf .*Tests.*_gta.exe c:\mg\aqs_test\medigraf .*Tests.*_gta.exe c:\mg\aqs_test\medigraf

</RunSettings>`