Test discovery lauches UWP executables

[djee-ms]

NUnit 3.12.0 with NUnit3TestAdapter 3.13.0 under Visual Studio 2019 16.3.8 The VS solution contains a mix of C++ projects, C# .NET Standard 2.0 libraries and .NET Core 3.0 apps, and some C# UWP apps.

During test discovery, the C# UWP app (executable) is started even though this is a GUI executable and is not referencing NUnit or any other testing package/assembly. Then discovery timeouts after 90 seconds.

I understand that UWP is not supported (#322), but this is both annoying (have to close the GUI app) and a loss of productivity (any test discovery is > 90 seconds). Is there any way to at least mark that UWP project as ignored for test discovery purpose? Or better, ignore it automatically?

[OsirisTerje]

Do you have the possibility to create a small repro project?

And yes, I have tried to compile the sln you refer to, and it doesn't compile.....

If you can reduce the problem to the essential, we might be able to help.

You can add filters for eliminating code, that is available in the Test Explorer search field, using the Fullname/FullyQualifiedName property, but I am not sure if that does all you need. It would be easier to really understand what triggers the run of the GUI, it doesn't make sense that it should happen during discovery.

[mikkelbu]

@djee-ms Does VS respect <IsTestProject>false</IsTestProject> in the project file? E.g. something like

  <PropertyGroup>
    <IsTestProject>false</IsTestProject>
  </PropertyGroup>

[djee-ms]

Hi,

@mikkelbu > I tried <IsTestProject>false</IsTestProject> but this has no effect. I added it to the first <PropertyGroup> block of the .csproj of the GUI application, is that what you expected? Is this property documented somewhere? Is it supposed to be picked up by the test adapter? I couldn't find any info about it.

@OsirisTerje > You likely didn't clone the submodules. The projects compile, we have CI green almost all the time. I will see if I can find some time to craft a smaller repro when I'm back at work next week, but we're a small team and are very busy. The usage scenario is being able to just press the "Run All Tests" button to check if a change broke something. The GUI app is started when we press that button in the Test Explorer panel, and the Output window of Visual Studio shows this corresponds to the start of the discovery. Then we need to wait for the 90s timeout before the tests actually start. This repro 100% of the time. Here's an example log; the GUI app starts around the time the very first line is output:

[03-Jan-20 3:05:25.836 PM] ---------- Discovery started ----------
Microsoft.VisualStudio.TestPlatform.ObjectModel.TestPlatformException: vstest.console process failed to connect to testhost process after 90 seconds. This may occur due to machine slowness, please set environment variable VSTEST_CONNECTION_TIMEOUT to increase timeout.
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.ThrowExceptionOnConnectionFailure(Int32 connTimeout)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyOperationManager.SetupChannel(IEnumerable`1 sources, String runSettings)
   at Microsoft.VisualStudio.TestPlatform.CrossPlatEngine.Client.ProxyDiscoveryManager.DiscoverTests(DiscoveryCriteria discoveryCriteria, ITestDiscoveryEventsHandler2 eventHandler)
[03-Jan-20 3:06:56.886 PM] ========== Discovery aborted: 0 tests found (0:01:30.9208358) ==========
[03-Jan-20 3:06:57.099 PM] ---------- Run started ----------
Multiple versions of same extension found. Selecting the highest version.
  GoogleTestAdapter.TestAdapter : 0.16.1.1369
GTA option 'DebugMode' does not have any effect any more - check your settings files and replace any occurence with new option 'OutputMode' as follows:
<DebugMode>False</DebugMode> => <OutputMode>Info</OutputMode>
<DebugMode>True</DebugMode> => <OutputMode>Verbose</OutputMode> (consider using Debug)
GTA option 'UseNewTestExecutionFramework' does not have any effect any more - check your settings files and replace any occurence with new option 'DebuggerKind' as follows:
<UseNewTestExecutionFramework>False</UseNewTestExecutionFramework> => <DebuggerKind>VsTestFramework</DebuggerKind>
<UseNewTestExecutionFramework>True</UseNewTestExecutionFramework> => <DebuggerKind>Native</DebuggerKind>
GTA option 'TimestampOutput' does not have any effect any more - check your settings files and replace any occurence with new option 'TimestampMode' as follows:
<TimestampOutput>False</TimestampOutput> => <TimestampMode>DoNotPrintTimestamp</TimestampMode>
<TimestampOutput>True</TimestampOutput> => <TimestampMode>PrintTimestamp</TimestampMode>
GTA option 'ShowReleaseNotes' is deprecated - check your settings files and remove any occurence.
Google Test Adapter: Test execution starting...
Running 19 tests...
Google Test execution completed, overall duration: 00:00:44.0894080.

================
The following warnings and errors occured during test execution (enable debug mode for more information):
GTA option 'DebugMode' does not have any effect any more - check your settings files and replace any occurence with new option 'OutputMode' as follows:
<DebugMode>False</DebugMode> => <OutputMode>Info</OutputMode>
<DebugMode>True</DebugMode> => <OutputMode>Verbose</OutputMode> (consider using Debug)
GTA option 'UseNewTestExecutionFramework' does not have any effect any more - check your settings files and replace any occurence with new option 'DebuggerKind' as follows:
<UseNewTestExecutionFramework>False</UseNewTestExecutionFramework> => <DebuggerKind>VsTestFramework</DebuggerKind>
<UseNewTestExecutionFramework>True</UseNewTestExecutionFramework> => <DebuggerKind>Native</DebuggerKind>
GTA option 'TimestampOutput' does not have any effect any more - check your settings files and replace any occurence with new option 'TimestampMode' as follows:
<TimestampOutput>False</TimestampOutput> => <TimestampMode>DoNotPrintTimestamp</TimestampMode>
<TimestampOutput>True</TimestampOutput> => <TimestampMode>PrintTimestamp</TimestampMode>
GTA option 'ShowReleaseNotes' is deprecated - check your settings files and remove any occurence.
NUnit Adapter 3.13.0.0: Test execution started
Running selected tests in F:\mrtk\mr-webrtc\bin\netstandard2.0\Debug\Microsoft.MixedReality.WebRTC.Tests.dll
   NUnit3TestExecutor converted 21 of 21 NUnit test cases
NUnit Adapter 3.13.0.0: Test execution complete
[03-Jan-20 3:08:07.567 PM] ========== Run finished: 40 tests run (0:01:10.4151451) ==========

[OsirisTerje]

@djee-ms Thanks! I'll try to clone the submodules too. A small[est] repro would however be awesome.

The IsTestProject is not used by the adapter, but it should have caused the TestHost to not send the assembly down to the adapter for testing.

I'm really curious to see how the discovery phase could trigger an executable to run though,.

[OsirisTerje]

@djee-ms I can now compile it. I didn't have SDK 10.0.15138 locally (and couldn't find it quickly for downloads), so upgraded to 18362.
I don't think the UWP app is started by the discovery of the NUnit adapter. To me it looks like it is started by the native tests when they are running. You mentioned discovery, but when I look at the test output it looks like it happens during execution.
Looking at the process when it runs, the TestAppUWP is started very early, before any of the other app processes starts up, incl tests. From the start event, I can see it is being started from the vstest.console.exe.

I can also see that the vstest.console is running just before the TestAppUwp is running, and no Now this doesn't exclude adapters from being involved, but it seems more likely that the testhost itself is starting the UWP app. That can also be seen in the process for the vstest.console.

Now, I am not at all sure what all this means, but it seems likely that the UWP app is started as part of the work done by vstest.console.

Perhaps @kendrahavens (MSFT, Test PG) knows more about this?

[djee-ms]

Thanks for the feedback! So it would be an issue with Visual Studio itself rather than the test adapter? I am not sure what happens when pressing the "Run All Tests" in the Test Explorer. @kendrahavens is there another place to report that issue if the NUnit test adapter is not involved?

[OsirisTerje]

I also see the following: And here one can see that the process 71160 which is the vstest.console is actually starting this one. And further up (a bit earlier in the chain) , it says that the UWP test app is an Activateable class, which is probably the reason it is being picked up and started, even if not intended.
Also note that "something" knows about a set of parameters to be used here when activating it. Not sure where that information comes from, but it must be from within the app itself, and added to the registry which is being read.

It would make sense if the app was stopped from being created by the test process using the IsTestProject attribute @mikkelbu mentions earlier in the thread.

Kendra will know what can be done here. I'll shoot her a mail too.

[OsirisTerje]

@djee-ms The PG confirmed back today that this is a known issue, reported here: https://developercommunity.visualstudio.com/content/problem/730446/when-i-run-unit-tests-it-launches-my-uwp-app.html and they are actively working on fixing it.

[djee-ms]

@OsirisTerje ok thanks, let's close this issue then, you cannot do anything about it. I left a comment and upvoted the VS issue.

[OsirisTerje]

This issue is fixed in Visual Studio 2019, update 16.5

[djee-ms]

Just tried and that looks fixed indeed, thanks @OsirisTerje.