search

csoltenborn / GoogleTestAdapter

Visual studio extension that adds support for the C++ testing framework Google Test.
Other
142 stars 102 forks source link

Test discovery fails randomly (when using solution settings file (?)) #332

Closed quxflux closed 3 years ago

quxflux commented 3 years ago

Hello I'm trying to get GoogleTestAdapter running with a solution settings file containing a <PathExtension> setting. However, I'm not sure whether the contents of the settings file do matter for this bug. The used VS version is 2019 16.11.2. The Bug can be reproduced at least with GoogleTestAdapter version v0.18.0 and the current version in master.

Occasionally when opening a solution following log is written to the Test output:

Log level is set to Informational (Default).
Test data store opened in 0,085 sec.
Failed to configure settings for runsettings plugin 'GoogleTestAdapterSettings' as it threw following exception:
'Object reference not set to an instance of an object.'
Please contact the plugin author.
========== Starting test discovery ==========
No settings file provided through env variable GTA_FALLBACK_SETTINGS
Warning: Using default settings.
Google Test Adapter: Test discovery starting...
Could not list test cases of executable 'C:\Users\****\source\repos\google_test_adapter_bug\x64\Release\some_google_test.exe': executing process failed with return code -1073741515
Command executed: 'C:\Users\****\source\repos\google_test_adapter_bug\x64\Release\some_google_test.exe --gtest_list_tests', working directory: 'C:\Users\****\source\repos\google_test_adapter_bug\x64\Release'
Command produced no output
Found 0 tests in executable C:\Users\****\source\repos\google_test_adapter_bug\x64\Release\some_google_test.exe
Test discovery completed, overall duration: 00:00:00.1483784

Because the solution settings file could not be loaded <PathExtension> will not be set, which in turn causes the test discovery to fail, since the test executable cannot find its required dependencies.

When Debugging the extension, it can be observed that an assertion on RunSettingsService.cs:126 is being triggered. This is caused by Package.GetGlobalService(typeof(DTE)) as DTE; resulting in null which will cause the following statements to fail. Stacktrace:

---- Assert Short Message ----
dte == null!
---- Assert Long Message ----

   at GoogleTestAdapter.TestAdapter.Settings.RunSettingsService.GetSolutionSettingsXmlFile()
   at GoogleTestAdapter.TestAdapter.Settings.RunSettingsService.GetValuesFromSolutionSettingsFile(RunSettingsContainer settingsContainer, ILogger logger)
   at GoogleTestAdapter.TestAdapter.Settings.RunSettingsService.AddRunSettings(IXPathNavigable runSettingDocument, IRunSettingsConfigurationInfo configurationInfo, ILogger logger)
   at Microsoft.VisualStudio.TestWindow.Controller.RunsettingsProvider.GenerateTestRunSettings(RunSettingConfigurationInfoState infoState, String runSettingsFilePath, IUserRunSettings runsettings, IEnumerable`1 runSettingsServices, IEnumerable`1 customSettingsImporters, RequestConfiguration requestConfiguration, ITestWindowTelemetry telemetryService)
   at Microsoft.VisualStudio.TestWindow.Core.Client.ContainerRunSettingsProvider.<GetRunAllSettingsMapAsync>d__3.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(Object stateMachine)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.VisualStudio.TestWindow.Core.Client.ContainerRunSettingsProvider.RunSettingsRetriever.<GetRunSettingsFileAsync>d__12.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(Object stateMachine)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(Action action, Boolean allowInlining, Task& currentTask)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetResult(TResult result)
   at Microsoft.VisualStudio.TestWindow.Core.Client.ContainerRunSettingsProvider.RunSettingsRetriever.<GetProjectRunSettingFileAsync>d__11.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.InvokeMoveNext(Object stateMachine)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.MoveNextRunner.Run()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<>c__DisplayClass4_0.<OutputAsyncCausalityEvents>b__0()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Runtime.CompilerServices.TaskAwaiter.<>c__DisplayClass11_0.<OutputWaitEtwEvents>b__0()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.ContinuationWrapper.Invoke()
   at System.Threading.Tasks.AwaitTaskContinuation.ExecuteWorkItemHelper()
   at System.Threading.Tasks.AwaitTaskContinuation.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

A solution which may be used to reproduce the error is attached. I'm a little bit lost here how to fix this. Does someone have an idea how to fix this issue?

google_test_adapter_bug.zip

Update The bug only seems to appear when the VS Test Explorer Pane is visible right at the startup of VS (which may cause the extension to be loaded to early before all dependent functionality is loaded)

csoltenborn commented 3 years ago

Sorry, just saw this... thanks for the great bug report!

From what I can see, your conclusions appear to be correct. Since you seem to be able to run GTA from source: Might it be an idea to add a "retry functionality" to the point where the DTE is received? I'm thinking about retrying for a couple of times (and e.g. once every second) in the hope that the DTE becomes available in time...

For what it's worth, a workaround might be to set the env variable as mentioned in the log output. This had been introduced because of a different (and now fixed) VS regression - the DTE is used to figure out the location of the solution settings file if I remember correctly, and by setting that variable you tell GTA where to find that file without reyling on the DTE object. I'm aware that this is not nice at all, but maybe better then having to live with the occasional problem.

quxflux commented 3 years ago

Thank you for your reply.

I've implemented the suggested retry mechanism (actually more like a brute-force while loop), which solves this particular error (so does the fallback file specified via the environment variable btw.). Unfortunately my overall goal was to get the test adapter running with 3rd party dependencies not present in the test targets $(OutDir) via placeholders in the PathExtension setting, as proposed in #170. There seem to be several places where the information which platform and configuration is selected does not get updated or populated in the first place properly. Two cases could be identified:

As the PATH variable does not get updated properly in these cases, loading the correct 3rd party dependencies fails and the test runners cannot be queried by the test adapter.

I gave up and will stick with the good ol' copy-all-the-dlls-to-the-output-directory method.

csoltenborn commented 3 years ago

Ok, thank for letting me know... I don't think that I have any influence on when platformName etc. are updated by VS - I just take what I get from the VS platform :-)

Closing this now - feel free to reopen...