Test discovery with custom working directory and command line parameters

[lysanntranvouez]

Is there a particular reason for the test discovery not passing on the command line parameters and working directory that can be configured for text execution?

I'm working on a project that

• loads test information from disk—and actually generates test cases via Google Test's Value-Parameterized Tests, and
• requires a command line option for google test to be initialized in the first place.

This breaks the test discovery of Google Test Adapter since it never finds any tests.

I could solve my issue if

• Either these things are passed on to the executable when discovering tests.
• Or there was a way to flag a project to always assume to have tests (skipping the current discovery process) and discovering tests on the fly by executing them.

Which of those approaches seems preferable? Other thoughts?

[csoltenborn]

I'm not sure whether I completely understand your problem yet...

• GTA does not mess with command line parameters - they can in fact be used by tests at test execution time. See the tests project of the SampleTests solution for a trivial example.
• The $(WorkingDir) placeholder can be used during test execution!
• Are you generating tests and then register them with Google Test, e.g. comparable to this?

My suggestion would be to get your tests running as desired on the console, and then post your exact invocations for test discovery (i.e., with parameter --gtest_list_tests) and test execution (GTA does exactly this, and then parses the respective output and reports it to the VS test framework). I then hopefully will have a better idea on how to help you...

One more thing: In general, make sure to load your test data within the appropriate Google Test setup/teardown facilities.

[lysanntranvouez]

My problem is not with test execution, but test discovery. The step where for every executable it runs it with the --gtest_list_tests command line option.

I need it to be run with that_program.exe -test -another_option --gtest_list_tests, and from a certain working directory. So I've set up the additional arguments using AdditionalTestExecutionParam in the solution.gta.runsettings configuration file.

It adds the arguments for test execution, but not for discovery (running it with that_program.exe --gtest_list_tests only instead).

Looking through the code, this appears to be the culprit:

https://github.com/csoltenborn/GoogleTestAdapter/blob/fb3ca8df290ba46ab4d54196e64e543c4424e1ad/GoogleTestAdapter/Core/TestCases/TestCaseFactory.cs#L166

string workingDir = new FileInfo(_executable).DirectoryName;
// ...
executor.ExecuteCommandBlocking(
    _executable,
    GoogleTestConstants.ListTestsOption,
    workingDir,
    _settings.GetPathExtension(_executable),
    lineAction);

It never uses the command line options or working directory from the settings for this step.

[csoltenborn]

That's true - in fact, I was not aware of the parameters, but usage of the $(WorkingDir) placeholder during discovery has already been fixed (see #200) - feel free to give this build a try.

Adding support for additional parameters shouldn't be to hard - would you be willing to test?

Having said that, I still would like to mention that your tests might suffer from a design smell. My understanding is that all initialization code should live within the test setup/teardown facilities of Google Test. What's the reason for the need of these parameters at discovery time? Are you indeed generating tests at runtime, and need file access for that?

[csoltenborn]

Feel free to give this version a shot, which contains a new option to provide parameters for test discovery...

[csoltenborn]

In case you want to give it a try, please use this build.

[lysanntranvouez]

Thanks! Just checking in to say that I unfortunately might not have opportunity this week to test it - something else came up. Super sorry about that.

(I’m still very interested though!)

[lysanntranvouez]

Thanks for implementing support for it!

I have tried the latest develop build, and yes, passing through the parameters to discovery works :)

---

I believe there is one bug with the parameters on test execution. If I set the AdditionalTestExecutionParam in the ProjectSettings they don't get applied when running the tests. Example output:

Visual Studio Version: VS2015
Google Test Adapter: Test execution starting...
Solution settings: AdditionalPdbs: '', AdditionalTestExecutionParam: '', BatchForTestSetup: '', BatchForTestTeardown: '', BreakOnFailure: False, CatchExceptions: True, DebugMode: True, KillProcessesOnCancel: False, MaxNrOfThreads: 8, NrOfTestRepetitions: 1, ParallelTestExecution: False, ParseSymbolInformation: True, PathExtension: '', PrintTestOutput: True, RunDisabledTests: False, ShowReleaseNotes: False, ShuffleTests: False, ShuffleTestsSeed: 0, SkipOriginCheck: False, TestDiscoveryRegex: '<redacted>', TestDiscoveryTimeoutInSeconds: 30, TestNameSeparator: '', TimestampOutput: False, TraitsRegexesAfter: {}, TraitsRegexesBefore: {}, UseNewTestExecutionFramework: True, WorkingDir: '$(ExecutableDir)'
No test case filter provided
Running 32 tests...
Settings for test executable 'C:\Users\lysann.schlegel\path\to\executable\my_executable.exe': AdditionalPdbs: '', AdditionalTestExecutionParam: '<my parameters>', BatchForTestSetup: '', BatchForTestTeardown: '', BreakOnFailure: False, CatchExceptions: True, DebugMode: True, KillProcessesOnCancel: False, MaxNrOfThreads: 8, NrOfTestRepetitions: 1, ParallelTestExecution: False, ParseSymbolInformation: True, PathExtension: '', PrintTestOutput: True, RunDisabledTests: False, ShowReleaseNotes: False, ShuffleTests: False, ShuffleTestsSeed: 0, SkipOriginCheck: False, TestDiscoveryRegex: '<redacted>', TestDiscoveryTimeoutInSeconds: 120, TestNameSeparator: '', TimestampOutput: False, TraitsRegexesAfter: {}, TraitsRegexesBefore: {}, UseNewTestExecutionFramework: True, WorkingDir: '$(ExecutableDir)'
>>>>>>>>>>>>>>> Output of command 'C:\Users\lysann.schlegel\path\to\executable\my_executable.exe --gtest_output="xml:C:\Users\lysann.schlegel\AppData\Local\Temp\tmp8D47.tmp" --gtest_catch_exceptions=1 --gtest_break_on_failure=0'

While if I set AdditionalTestExecutionParam in the SolutionSettings it works as expected:

Visual Studio Version: VS2015
Google Test Adapter: Test execution starting...
Solution settings: AdditionalPdbs: '', AdditionalTestExecutionParam: '<my parameters>', BatchForTestSetup: '', BatchForTestTeardown: '', BreakOnFailure: False, CatchExceptions: True, DebugMode: True, KillProcessesOnCancel: False, MaxNrOfThreads: 8, NrOfTestRepetitions: 1, ParallelTestExecution: False, ParseSymbolInformation: True, PathExtension: '', PrintTestOutput: True, RunDisabledTests: False, ShowReleaseNotes: False, ShuffleTests: False, ShuffleTestsSeed: 0, SkipOriginCheck: False, TestDiscoveryRegex: '<redacted>', TestDiscoveryTimeoutInSeconds: 120, TestNameSeparator: '', TimestampOutput: False, TraitsRegexesAfter: {}, TraitsRegexesBefore: {}, UseNewTestExecutionFramework: True, WorkingDir: '$(ExecutableDir)'
No test case filter provided
Running 32 tests...
No settings configured for test executable 'C:\Users\lysann.schlegel\path\to\executable\my_executable.exe'; running with solution settings: AdditionalPdbs: '', AdditionalTestExecutionParam: '<my parameters>', BatchForTestSetup: '', BatchForTestTeardown: '', BreakOnFailure: False, CatchExceptions: True, DebugMode: True, KillProcessesOnCancel: False, MaxNrOfThreads: 8, NrOfTestRepetitions: 1, ParallelTestExecution: False, ParseSymbolInformation: True, PathExtension: '', PrintTestOutput: True, RunDisabledTests: False, ShowReleaseNotes: False, ShuffleTests: False, ShuffleTestsSeed: 0, SkipOriginCheck: False, TestDiscoveryRegex: '<redacted>', TestDiscoveryTimeoutInSeconds: 120, TestNameSeparator: '', TimestampOutput: False, TraitsRegexesAfter: {}, TraitsRegexesBefore: {}, UseNewTestExecutionFramework: True, WorkingDir: '$(ExecutableDir)'
>>>>>>>>>>>>>>> Output of command 'C:\Users\lysann.schlegel\path\to\executable\my_executable.exe --gtest_output="xml:C:\Users\lysann.schlegel\AppData\Local\Temp\tmpF4B7.tmp" --gtest_catch_exceptions=1 --gtest_break_on_failure=0 <my parameters>'

And note that it's working fine during discovery.

Just looking through the code, I think the issue might be here:

https://github.com/csoltenborn/GoogleTestAdapter/blob/60355b914ae680804d874b8d1e9db10bb6280c1d/GoogleTestAdapter/Core/Runners/SequentialTestRunner.cs#L50

It's extracting workingDir and userParameters before/outside ExecuteWithSettingsForExecutable.

I would try to fix it myself and test it, but I really have no clue how to set up a development environment for VS extensions :/

I believe this issue has been introduced during the refactorings around the addition of this feature. Let me know if you want me to create a separate issue for this.

[csoltenborn]

Thanks for your feedback, Lysann! And you appear to be right with your new issue as well as the proposed fix :-) I have just pushed the according change - feel free to give it another try...

In general, if you want to try something with GTA, have a look at the setup description - it's in fact pretty easy!

[lysanntranvouez]

Works :)

Thanks for the quick feedback and fix!