vstest console wrapper seems to not receive DiscoveryComplete event since 17.2.0

[rouke-broersma]

Description

In stryker-net we have not been able to upgrade the TestPlatform since 17.2.0 due to an infinite hang during test discovery. We had not been able to investigate properly because it was only failing on azure devops for some reason. I have finally had time to figure out why we weren't getting proper logs in the pipeline, and have found that the reason for the hang is that vstest.console.exe does seem to send the DiscoveryComplete but our wrapper is not receiving it.

Steps to reproduce

https://github.com/stryker-mutator/stryker-net/pull/2221 https://dev.azure.com/stryker-mutator/Stryker/_build/results?buildId=12695&view=results

Diagnostic logs

TestDiscoverer_VsTest-log.txt

Environment

Azure Pipelines hosted agent - windows latest dotnet framework 4.8

[nohwnd]

@rouke-broersma have you been able to ever repro locally?

[rouke-broersma]

I haven't tried recently, but as far as I remember from past attempts no, it only fails in the pipeline. I'll run it locally right now and i'll let you know.

Edit;

It seems that I can now repro this locally (since at least 17.4). Perhaps this is actually a different problem than we experienced before (haven't tested all changes locally and haven't closely looked at the build status because the PR had been failing since 17.2.0).

[rouke-broersma]

local results:

TestDiscoverer_VsTest-log.txt

I ran this with a breakpoint here: https://github.com/stryker-mutator/stryker-net/blob/32de8379a0201a000270f23c3ca87fdec41b3772/src/Stryker.Core/Stryker.Core/TestRunners/VsTest/DiscoveryEventHandler.cs#L38

But the breakpoint was not reached even though the logs say that the DiscoveryCompleted is logged:

TestDiscoveryManager.DiscoveryComplete: Called DiscoveryComplete callback.
DiscoveryRequest.Dispose: Starting.
DiscoveryRequest.Dispose: Completed.
TestRequestManager.DiscoverTests: Discovery tests completed.

The HandleDiscoveredTests callback is also not executed. Seems like our DiscoveryEventHandler is not being loaded?

We are passing it to the vstest console wrapper DiscoverTests here: https://github.com/stryker-mutator/stryker-net/blob/32de8379a0201a000270f23c3ca87fdec41b3772/src/Stryker.Core/Stryker.Core/TestRunners/VsTest/VsTestContextInformation.cs#L203

[nohwnd]

I've updated the packages to 17.5.0 and I am running .CLI directly from your repo against your xunit test project in your integration tests.

diff --git a/src/Directory.Packages.props b/src/Directory.Packages.props
index b7e74947..01eee8fe 100644
--- a/src/Directory.Packages.props
+++ b/src/Directory.Packages.props
@@ -22,15 +22,15 @@
     <PackageVersion Include="Spectre.Console.Analyzer" Version="0.46.0" />
     <PackageVersion Include="System.Net.Http.Json" Version="7.0.0" />
     <PackageVersion Include="Microsoft.CodeAnalysis.CSharp" Version="4.5.0" />
-    <PackageVersion Include="Microsoft.TestPlatform" Version="17.2.0" />
-    <PackageVersion Include="Microsoft.TestPlatform.ObjectModel" Version="17.2.0" />
-    <PackageVersion Include="Microsoft.TestPlatform.Portable" Version="17.2.0" />
-    <PackageVersion Include="Microsoft.TestPlatform.TranslationLayer" Version="17.2.0" />
+    <PackageVersion Include="Microsoft.TestPlatform" Version="17.5.0" />
+    <PackageVersion Include="Microsoft.TestPlatform.ObjectModel" Version="17.5.0" />
+    <PackageVersion Include="Microsoft.TestPlatform.Portable" Version="17.5.0" />
+    <PackageVersion Include="Microsoft.TestPlatform.TranslationLayer" Version="17.5.0" />
     <PackageVersion Include="TestableIO.System.IO.Abstractions.Wrappers" Version="19.1.18" />
     <PackageVersion Include="Microsoft.Extensions.Logging" Version="7.0.0" />
   </ItemGroup>
   <ItemGroup>
-    <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
+    <PackageVersion Include="Microsoft.NET.Test.Sdk" Version="17.5.0" />
     <PackageVersion Include="coverlet.collector" Version="3.2.0" />
     <PackageVersion Include="coverlet.msbuild" Version="3.2.0" />
     <PackageVersion Include="Moq" Version="4.18.4" />

I am hitting the breakpoint on DiscoveryEventHandler. Any additional tips on how to repro?

[rouke-broersma]

The xunit project is dotnet core. It's working fine for dotnet core (we use vstest.console.dll). It's broken on dotnet framework (we use vstest.console.exe) The dotnet framework integration test project is here: https://github.com/stryker-mutator/stryker-net/tree/master/integrationtest/TargetProjects/NetFramework/FullFrameworkApp.Test

You can checkout this branch: https://github.com/stryker-mutator/stryker-net/tree/renovate/vstest-monorepo It has the updates applied.

I use this launch profile:

    "stryker-it-framework": {
      "commandName": "Project",
      "commandLineArgs": "-V trace -L",
      "workingDirectory": "C:\\git\\stryker\\stryker-net\\integrationtest\\TargetProjects\\NetFramework\\FullFrameworkApp.Test"
    },

[rouke-broersma]

Did you manage to repro with these instructions?

[rouke-broersma]

@nohwnd If I can help in any way let me know :)

[nohwnd]

I've spent a month trying to move our repo to new infra. Now I can get back to this.

[rouke-broersma]

No worries, I appreciate whatever time you can find! Just making sure there isn't something I can do to help or that I'm not the one blocking a successful investigation.

[rouke-broersma]

Unrelated changes in https://github.com/stryker-mutator/stryker-net/pull/2400 seem to have fixed the issue for some reason, we are no longer blocked. However I want to note that we have been blocked with the exact same behavior in the past before.