Closed ericstj closed 2 weeks ago
Tagging subscribers to this area: @dotnet/area-system-collections See info in area-owners.md if you want to be subscribed.
Author: | ericstj |
---|---|
Assignees: | - |
Labels: | `area-System.Collections` |
Milestone: | - |
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries See info in area-owners.md if you want to be subscribed.
Author: | ericstj |
---|---|
Assignees: | - |
Labels: | `area-Infrastructure-libraries`, `untriaged` |
Milestone: | - |
Someone needs to debug this. This likely also impacts VS Text Explorer which uses the VSTest runner underneath. There are settings to get more verbose logs returned by VSTest.
One more thought, that could be related to the recent xunit 2.5.x upgrade.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.757, 3877444491007, vstest.console.dll, MetadataReaderExtensionsHelper: Discovering extensions inside assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.8.21101, Culture=neutral, PublicKeyToken=31bf3856ad364e35' file path 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Diagnostics.NETCore.Client.dll'
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444578798, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolving assembly.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444580691, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Searching in: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444582066, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Bcl.AsyncInterfaces.dll', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583023, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Bcl.AsyncInterfaces.exe', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583316, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Searching in: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583907, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Microsoft.Bcl.AsyncInterfaces.dll', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444584450, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Microsoft.Bcl.AsyncInterfaces.exe', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444584752, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Failed to load assembly.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444585113, vstest.console.dll, CurrentDomainAssemblyResolve: Resolving assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444585764, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolving assembly.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586390, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolved from cache.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586674, vstest.console.dll, CurrentDomainAssemblyResolve: Resolving assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586988, vstest.console.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.768, 3877444596875, vstest.console.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Warning: 0 : 25352, 1, 2023/10/30, 19:39:45.783, 3877444748536, vstest.console.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.8.21101, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
at System.Reflection.RuntimeModule.GetTypes()
at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos, String filePath) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginDiscoverer.cs:line 158
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
File name: 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'Microsoft.Bcl.AsyncInterfaces, Culture=neutral, PublicKeyToken=null'
at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginCache.CurrentDomainAssemblyResolve(Object sender, AssemblyResolveEventArgs args) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginCache.cs:line 513
at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName)
at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName)
at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName)
--diag:log.txt
This is not what you think it is.
Are you're linking against xunit.runner.utility for some reason? If so, then you're frozen to using whatever version of xunit.runner.utility the VS adapter uses (because VSTest demands all dependencies be copied into the output folder). 99%+ of users don't do this so this isn't a general problem.
The only reason to link against xunit.runner.utility is because you're creating your own runner. Tests should never link against this unless they're forced to because they're testing the runner you're writing.
For a bit of architectural background...
xUnit.net splits what it does into two separate sections which do not directly interact except via very strict contracts. There are "runner" and "execution" halves of the system. In v1/v2 when using .NET Framework, these two halves can be optionally separated by an app domain boundary (and in v2 .NET Core, they live in the same process). In v3 against all target platforms, they're separated by a process boundary.
xunit.core
and xunit.execution.*
(and usually xunit.assert
, though that's optional).xunit.runner.utility.*
(and maybe other runner related things).xunit.abstractions
. It has been frozen since we shipped 2.0 (we have shipped newer versions of the NuGet assembly just to add extra targets as 2.0 was back in PCL259 days, but the DLL itself remains unchanged since we shipped it).By keeping this strong separation and frozen contracts, we have both backward compatibility (newer runners can run older tests) and major forward compatibility (older v2 runners can run newer v2 tests). In point of fact, I just was able to successfully run tests linked against core framework 2.5.3 with the VSTest adapter 2.0.0, which is 8.5 years old and targeted net20
, wpa81
, and win8
. Obviously .NET Core didn't even exist back then, so it wouldn't be able to run a platform it doesn't know about, but .NET Framework is not a problem.
If your tests link against xunit.runner.utility.*
, you're bringing the runner half of things where it normally doesn't live or belong. This is unavoidable if you're writing your own runner, and want to test that runner (we obviously do this, but it should be very very rare). Normally this wouldn't be a problem except when you're using VSTest, because there is a requirement placed by VSTest that we copy the VSTest adapter into your output folder, which of course means we also have to copy the runner utility libraries (xunit.runner.utility.*
and xunit.runner.reporters.*
). This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you're in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you're using.
In addition, there is no guarantee that any particular version of xunit.runner.visualstudio
will use any particular version of xunit.runner.utility
. We shipped 2.5.3 of the former at the same time as 2.5.2 of the latter, but even simultaneous shipment isn't guaranteed (and in fact they lived very separate lives back when @clairernovotny owned the VSTest adapter).
When I shipped core framework 2.5.3, it had no runner utility changes that would've necessitated bringing in a new version for the VSTest adapter, so I didn't bother to ship a new one. It appears that when I ship the next core framework drop (which I'm planning to number 2.6.0), this will still be true, so again I'm not likely to ship another VSTest adapter version.
Wow, that's quite a detailed comment. Thank you for spending your time on writing this up. I just reproduced this locally and I don't see our test assemblies or test helper assemblies linking against any of the xunit.runner.*
assemblies.
What I see is that because of us wanting to support both xunit.console and VSTest, we bring in xunit.runner.utility.net452.dll
from the xunit.console and explicitly exclude the one from xunit.runner.visualstudio which results in that version collision.
This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you're in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you're using.
Exactly that.
This begs the question why we ever chose a design that made us copy the xunit.console runner into the test app's output directory. While VSTest requires that, xunit.console shouldn't.
Yeah, that operation makes no sense to me. You were basically just waiting to explode. 😁
Each runner has to bring along their own copy of xunit.runner.utility.*
that they're linked against. It's just the VSTest adapter which insists on copying it into your output folder; all the other runners just live elsewhere and are invoked by hand, so the assemblies are never mixed together.
To be even more explicit: xunit.console would never use xunit.runner.utility.*
from the test's folder. It would need to live along side xunit.console.exe
so that .NET Framework could do the appropriate implicit loading. If you're copying the contents of xunit.console's NuGet package into your test output folder, you should stop doing that. It's unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages
or some equivalent).
you should stop doing that. It's unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages or some equivalent).
Just to share some context. We invoke our tests in CI on different machines than the build agents. We use Helix
for that, which is a distributed machine testing orchestration infrastructure. Because of that, we can't rely on the package cache. Instead we should have just sent that dependency as an additional "payload" (dependency) to that helix test agent.
Unfortunately our infrastructure here has become quite complex with so many different app models and ways of testing that this change will probably require some time.
But for what is worth, this is indeed our own fault. Thanks for help diagnosing the issue :)
Good luck! 😄
@ViktorHofer did you have a suggestion for what needs to happen to fix this?
I gather that by copying xunit runner binaries over to the output directory this is somehow causing the test-discoverer to fail before it gets a chance to examine our test assembly, is that correct?
The gist (IIRC) is to not copy xunit.console's assemblies into the output directory and just invoke it from the package cache (as intended). Only copy it into the output directory when archiving the test assemblies for helix submissions (property: /p:ArchiveTests=true
).
xunit.runner.visualstudio is designed to be copied into the app output directory but xunit.console isn't. So avoiding the latter makes sense on a local machine.
xunit.runner.visualstudio is designed to be copied into the app output directory
That makes it sound like we do something special here. We don't. The copy is an unfortunate requirement from the VSTest team; if we don't copy the DLLs into the output directory, then whoever runs dotnet test
(or equivalent) needs to specify the location of the test adapter path on disk via an additional command line argument.
If it helps, we could add an MSBuild property that we look for to prevent the file copies. They would need to stay on by default but if you know what you're doing, skipping the copies could make sense.
I think this really is an issue on our side and needs a fix in our test infrastructure. The xunit.console copying is just wrong and also in-efficient. I should have done this differently in the past.
Yeah, other repos don't copy and bundle the runner package as a correlation payload. It's not so hard to make that work, but it does require changes in a few places.
One thing that was odd to me is that we're saying that our binaries are self-consistent and would work if we merely delete files that aren't needed. That tells me we might also have a bug / design-change ask for the dotnet test
scenario to not halt when it encounters binding problems in other assemblies in the output directory.
It's not so hard to make that work, but it does require changes in a few places.
I was suggesting something simpler for a quick fix. Keep the xunit.console in the app output directory but only when ArchiveTests=true
. Otherwise (locally) use the xunit.console from the nuget cache.
@akoeplinger found other places that still rely on xunit.console in the output directory.
it looks like https://github.com/dotnet/runtime/pull/96826 is breaking a couple things
- A couple wasm projects set BundleXunitRunner but now they don't get the xunit.console.dll locally unless they set ArchiveTests=true
- It also breaks the enterprise-linux pipeline since it uses dotnet build /t:Test instead of dotnet test: https://dev.azure.com/dnceng-public/public/_build/results?buildId=537464&view=logs&j=03b95830-93c2-599b-7e4d-1efb940adc28&t=deab5cac-2f83-5667-2973-63a6919b97de
The latter seems more problematic since we document /t:Test in a couple places and I know many people are still in the habit of using it vs dotnet test and in some cases you don't want the vstest runner
It seems we have some more places to root out where we expect the runner in the output directory - those need to either be updated to pick up the runner from the package, or opt-in to copying.
The latter seems more problematic since we document /t:Test in a couple places and I know many people are still in the habit of using it vs dotnet test and in some cases you don't want the vstest runner
dotnet build /t:Test
should still work with https://github.com/dotnet/runtime/pull/96826. What are we missing here?
Moving this back into 9.0.0 as this makes debugging .NET Framework tests with dotnet test
harder. We can move it back to 10.0.0 if we don't get to it though.
Description
When running
dotnet test
on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one.Reproduction Steps
build clr+libs -rc Release cd src\libraries\System.Collections.Immutable\tests dotnet test -f:net462
Expected behavior
Tests run.
Actual behavior
No tests run:
Regression?
I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test.
Known Workarounds
Directly invoke the
test
target.Configuration
No response
Other information
No response