xunit / visualstudio.xunit

VSTest runner for xUnit.net (for Visual Studio Test Explorer and dotnet test)
https://xunit.net/
Other
147 stars 80 forks source link

xunit could cause the VS test discovery process to hang #291

Closed AbhitejJohn closed 2 years ago

AbhitejJohn commented 2 years ago

The setting of the Finished event here does not account for exceptions in the method call above. This was causing the discovery with VS Test Platform to never complete because xunit does not return back a discovery complete message. In our specific case the problem was that a stale binary referenced an older xunit.core assembly (2.4.1-pre.build.4059, which interestingly had a higher assembly version than the GA release, 2.4.1 (which is why binding redirection entries didn't help as well)). This was causing an assembly load failure at runtime with the following stack

System.IO.FileLoadException
  HResult=0x80131040
  Message=Could not load file or assembly 'xunit.core, Version=2.4.1.4059, Culture=neutral, PublicKeyToken=8d05b1bb7a6fdb6c' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)

mscorlib.dll!System.Reflection.RuntimeAssembly.GetType(string name, bool throwOnError, bool ignoreCase) Unknown
    mscorlib.dll!System.Reflection.Assembly.GetType(string name)    Unknown
    xunit.execution.desktop.dll!Xunit.Sdk.SerializationHelper.GetType(string assemblyName, string typeName) Line 189    C#
>   xunit.execution.desktop.dll!Xunit.Sdk.ExtensibilityPointFactory.GetTraitDiscoverer(Xunit.Abstractions.IMessageSink diagnosticMessageSink, Xunit.Abstractions.IAttributeInfo traitDiscovererAttribute) Line 199  C#
    xunit.execution.desktop.dll!Xunit.Sdk.XunitTestCase.Initialize() Line 134   C#
    xunit.execution.desktop.dll!Xunit.Sdk.TestMethodTestCase.UniqueID.get() Line 184    C#
    xunit.execution.desktop.dll!Xunit.Sdk.TestCaseDescriptorFactory.TestCaseDescriptorFactory(object discovererObject, object testCasesObject, object callbackObject) Line 33   C#
    [Native to Managed Transition]  
    [Managed to Native Transition]  
    mscorlib.dll!System.RuntimeType.CreateInstanceImpl(System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes, ref System.Threading.StackCrawlMark stackMark)  Unknown
    mscorlib.dll!System.Activator.CreateInstance(System.Type type, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes) Unknown
    mscorlib.dll!System.Activator.CreateInstance(string assemblyString, string typeName, bool ignoreCase, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes, System.Security.Policy.Evidence securityInfo, ref System.Threading.StackCrawlMark stackMark) Unknown
    mscorlib.dll!System.Activator.CreateInstance(string assemblyName, string typeName, bool ignoreCase, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes, System.Security.Policy.Evidence securityInfo)  Unknown
    mscorlib.dll!System.AppDomain.CreateInstance(string assemblyName, string typeName, bool ignoreCase, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes, System.Security.Policy.Evidence securityAttributes)    Unknown
    mscorlib.dll!System.AppDomain.CreateInstanceAndUnwrap(string assemblyName, string typeName, bool ignoreCase, System.Reflection.BindingFlags bindingAttr, System.Reflection.Binder binder, object[] args, System.Globalization.CultureInfo culture, object[] activationAttributes, System.Security.Policy.Evidence securityAttributes)   Unknown
    [AppDomain (testhost.net472.x86.exe, #1) -> AppDomain (Microsoft.CodeAnalysis.CSharp.CommandLine.UnitTests, #87)] 
    xunit.runner.utility.net452.dll!Xunit.AppDomainManager_AppDomain.CreateObject<object>(System.Reflection.AssemblyName assemblyName, string typeName, object[] args) Line 78  C#
    xunit.runner.utility.net452.dll!Xunit.Xunit2Discoverer.GetTestCaseDescriptors(System.Collections.Generic.List<Xunit.Abstractions.ITestCase> testCases, bool includeSerialization) Line 202  C#
    xunit.runner.utility.net452.dll!Xunit.XunitFrontController.GetTestCaseDescriptors(System.Collections.Generic.List<Xunit.Abstractions.ITestCase> testCases, bool includeSerialization) Line 184  C#
    xunit.runner.visualstudio.testadapter.dll!Xunit.Runner.VisualStudio.VsDiscoverySink.SendExistingTestCases() Line 204    C#
    xunit.runner.visualstudio.testadapter.dll!Xunit.Runner.VisualStudio.VsDiscoverySink.HandleTestCaseDiscoveryMessage(Xunit.MessageHandlerArgs<Xunit.Abstractions.ITestCaseDiscoveryMessage> args) Line 184    C#
    xunit.runner.utility.net452.dll!MessageSinkMessageExtensions.Dispatch<Xunit.Abstractions.ITestCaseDiscoveryMessage>(Xunit.Abstractions.IMessageSinkMessage message, System.Collections.Generic.HashSet<string> messageTypes, Xunit.MessageHandler<Xunit.Abstractions.ITestCaseDiscoveryMessage> callback) Line 39   C#
    xunit.runner.utility.net452.dll!Xunit.DiscoveryEventSink.OnMessageWithTypes(Xunit.Abstractions.IMessageSinkMessage message, System.Collections.Generic.HashSet<string> typeNames) Line 27   C#
    xunit.runner.visualstudio.testadapter.dll!Xunit.Runner.VisualStudio.VsDiscoverySink.Xunit.IMessageSinkWithTypes.OnMessageWithTypes(Xunit.Abstractions.IMessageSinkMessage message, System.Collections.Generic.HashSet<string> messageTypes) Line 197    C#
    xunit.runner.utility.net452.dll!Xunit.MessageSinkAdapter.OnMessageWithTypes(Xunit.Abstractions.IMessageSinkMessage message, System.Collections.Generic.HashSet<string> messageTypes) Line 50    C#

Although the stack above points to HandleTestCaseDiscoveryMessage the same happens when HandleDiscoveryCompleteMessage is triggered as well.

Logging this issue to help make the discovery process a little more reliable.

clairernovotny commented 2 years ago

Thanks @AbhitejJohn - any chance you could submit a PR with a fix?

AbhitejJohn commented 2 years ago

ah sure, sorry I couldn't get back to this earlier. I can put up a PR this week.