The latest `xunit.runner.visualstudio` does not work with mono

[0xced]

Tested on macOS (both 10.15.7 on x64 and 13.6.6 on amd64) with mono 6.12.0

Reproduction steps on the current main branch (commit d4c272148d87c15d9f990f04058445541f152cf5)

cd test/test.v2/
dotnet test

Output:

Determining projects to restore... Restored ~/visualstudio.xunit/test/test.v2/test.v2.csproj (in 674 ms). 1 of 2 projects are up-to-date for restore. xunit.runner.visualstudio -> ~/visualstudio.xunit/src/xunit.runner.visualstudio/bin/Debug/net472/xunit.runner.visualstudio.testadapter.dll xunit.runner.visualstudio.testadapter -> ~/visualstudio.xunit/src/xunit.runner.visualstudio/bin/Debug/net472/merged/xunit.runner.visualstudio.testadapter.dll xunit.runner.visualstudio -> ~/visualstudio.xunit/src/xunit.runner.visualstudio/bin/Debug/net6.0/xunit.runner.visualstudio.testadapter.dll xunit.runner.visualstudio.testadapter -> ~/visualstudio.xunit/src/xunit.runner.visualstudio/bin/Debug/net6.0/merged/xunit.runner.visualstudio.testadapter.dll test.v2 -> ~/visualstudio.xunit/test/test.v2/bin/Debug/net472/test.v2.dll test.v2 -> ~/visualstudio.xunit/test/test.v2/bin/Debug/net6.0/test.v2.dll Test run for ~/visualstudio.xunit/test/test.v2/bin/Debug/net472/test.v2.dll (.NETFramework,Version=v4.7.2) Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. No test is available in ~/visualstudio.xunit/test/test.v2/bin/Debug/net472/test.v2.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:. Test run for ~/visualstudio.xunit/test/test.v2/bin/Debug/net6.0/test.v2.dll (.NETCoreApp,Version=v6.0) Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. [xUnit.net 00:00:00.31] Tests.Skipped [SKIP] [xUnit.net 00:00:00.35] Tests.ConditionallyFailing(value: 1) [FAIL] Skipped Tests.Skipped [1 ms] Failed Tests.ConditionallyFailing(value: 1) [2 ms] Error Message: Assert.Equal() Failure: Values differ Expected: 0 Actual: 1 Stack Trace: at Tests.ConditionallyFailing(Int32 value) in ~/visualstudio.xunit/test/test.v2/Tests.cs:line 11

Failed! - Failed: 1, Passed: 2, Skipped: 1, Total: 4, Duration: 39 ms - test.v2.dll (net6.0)

I did investigate and I stumbled on a swallowed exception.

[xUnit.net 00:00:00.09] Catastrophic failure: System.TypeLoadException: VTable setup of type Xunit.Runner.VisualStudio.VisualStudioSourceInformationProvider failed
  at System.Runtime.CompilerServices.AsyncTaskMethodBuilder.Start[TStateMachine] (TStateMachine& stateMachine) [0x0002c] in <207d12884485488d9288cc8cbf474289>:0 
  at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssembly (Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle, Xunit.Runner.VisualStudio.LoggerHelper logger, Xunit.Runner.VisualStudio.TestPlatformContext testPlatformContext, Xunit.Runner.VisualStudio.RunSettings runSettings, Xunit.Sdk.IMessageSink reporterMessageHandler, Xunit.Runner.VisualStudio.AssemblyRunInfo runInfo) [0x00054] in <331c18b0dc244684aa85d03224f2ac22>:0 
  at Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass22_0.<RunTestsInAssemblyAsync>b__0 () [0x0001b] in <331c18b0dc244684aa85d03224f2ac22>:0 
  at Xunit.Runner.VisualStudio.VsTestRunner.RunTestsInAssemblyAsync (Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle, Xunit.Runner.VisualStudio.LoggerHelper logger, Xunit.Runner.VisualStudio.TestPlatformContext testPlatformContext, Xunit.Runner.VisualStudio.RunSettings runSettings, Xunit.Sdk.IMessageSink reporterMessageHandler, Xunit.Runner.VisualStudio.AssemblyRunInfo runInfo) [0x00066] in <331c18b0dc244684aa85d03224f2ac22>:0 
  at Xunit.Runner.VisualStudio.VsTestRunner+<>c__DisplayClass20_1.<RunTests>b__6 (Xunit.Runner.VisualStudio.AssemblyRunInfo runInfo) [0x00000] in <331c18b0dc244684aa85d03224f2ac22>:0 
  at System.Collections.Generic.List`1[T].ForEach (System.Action`1[T] action) [0x0001e] in <207d12884485488d9288cc8cbf474289>:0 
  at Xunit.Runner.VisualStudio.VsTestRunner.RunTests (Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle, Xunit.Runner.VisualStudio.LoggerHelper logger, Xunit.Runner.VisualStudio.TestPlatformContext testPlatformContext, Xunit.Runner.VisualStudio.RunSettings runSettings, System.Func`1[TResult] getRunInfos) [0x0031f] in <331c18b0dc244684aa85d03224f2ac22>:0 
  at Xunit.Runner.VisualStudio.VsTestRunner.RunTests (Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IRunContext runContext, Microsoft.VisualStudio.TestPlatform.ObjectModel.Adapter.IFrameworkHandle frameworkHandle, Xunit.Runner.VisualStudio.LoggerHelper logger, Xunit.Runner.VisualStudio.TestPlatformContext testPlatformContext, Xunit.Runner.VisualStudio.RunSettings runSettings, System.Func`1[TResult] getRunInfos) [0x003eb] in <331c18b0dc244684aa85d03224f2ac22>:0 

Note that I had to modify VsTestRunner.cs like this to be able to see the swallowed exception.

diff --git a/src/xunit.runner.visualstudio/VsTestRunner.cs b/src/xunit.runner.visualstudio/VsTestRunner.cs
index d7cd4cf..223c1e2 100644
--- a/src/xunit.runner.visualstudio/VsTestRunner.cs
+++ b/src/xunit.runner.visualstudio/VsTestRunner.cs
@@ -634,7 +634,7 @@ static void PrintHeader(LoggerHelper loggerHelper)
        {
            var @event = new ManualResetEvent(initialState: false);

-           ThreadPool.QueueUserWorkItem(async _ =>
+           Task.Run(async () =>
            {
                try
                {
@@ -644,7 +644,7 @@ static void PrintHeader(LoggerHelper loggerHelper)
                {
                    @event.Set();
                }
-           });
+           }).GetAwaiter().GetResult();

            return @event;
        }

Now, the TypeLoadException itself looks like a bug in mono.

System.TypeLoadException: VTable setup of type Xunit.Runner.VisualStudio.VisualStudioSourceInformationProvider failed

There's even an open issue about how this error message is not useful enough to diagnose what the underlying problem is. 😞

At first I thought it might be because VisualStudioSourceInformationProvider is using a primary constructor. So I tried to change it to an explicit constructor but that was not the problem. Now I suspect it might be because ISourceInformationProvider.GetSourceInformation() returns a named tuple but that's just a guess and it could be something else entirely.

I think it would be worth trying to change the return type of ISourceInformationProvider.GetSourceInformation() to a struct or class and see if that works around the TypeLoadException. If not then a bug should be filed to the mono repository but the current reproduction is a bit overwhelming I think.

[bradwilson]

As far as I can tell, there will be no bug fixes in Mono. It is a "complete" (or, if you prefer a slightly more negative term, "dead") project.

I'll take a look and see if that change makes a difference.

[bradwilson]

Your theory was correct, it was the tuple return type causing the failure. I'll have a build available shortly.

[bradwilson]

The fix is in 3.0.0-pre.24, which is now available on NuGet.

https://xunit.net/releases/visualstudio/3.0.0-pre.24

[0xced]

Awesome, thanks for fixing it and releasing a new version! 🎉