search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.92k stars 4.64k forks source link

`dotnet test` is not running tests for .NETFramework targets #94183

Closed ericstj closed 2 weeks ago

ericstj commented 10 months ago

Description

When running dotnet test on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one.

Reproduction Steps

build clr+libs -rc Release cd src\libraries\System.Collections.Immutable\tests dotnet test -f:net462

Expected behavior

Tests run.

    Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod,
   method display options = None)
    Discovered:  System.Collections.Immutable.Tests (found 4008 of 4123 test cases)
    Starting:    System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16)

Actual behavior

No tests run:

Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2)
Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

Regression?

I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test.

Known Workarounds

Directly invoke the test target.

\src\dotnet\runtime.dotnet\dotnet build /t:test -f:net462

Configuration

No response

Other information

No response

ghost commented 10 months ago

Tagging subscribers to this area: @dotnet/area-system-collections See info in area-owners.md if you want to be subscribed.

Issue Details
### Description When running `dotnet test` on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one. ### Reproduction Steps build clr+libs -rc Release cd src\libraries\System.Collections.Immutable\tests dotnet test -f:net462 ### Expected behavior Tests run. ``` Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod, method display options = None) Discovered: System.Collections.Immutable.Tests (found 4008 of 4123 test cases) Starting: System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16) ``` ### Actual behavior No tests run: ``` Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2) Starting test execution, please wait... A total of 1 test files matched the specified pattern. No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. ``` ### Regression? I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test. ### Known Workarounds Directly invoke the `test` target. > \src\dotnet\runtime\.dotnet\dotnet build /t:test -f:net462 ### Configuration _No response_ ### Other information _No response_
Author: ericstj
Assignees: -
Labels: `area-System.Collections`
Milestone: -
ghost commented 10 months ago

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries See info in area-owners.md if you want to be subscribed.

Issue Details
### Description When running `dotnet test` on a test project in dotnet/runtime that targets .NETFramework I expect tests to run, but they do not. Seems like we have lots of other issues mentioning this, but I couldn't find an active one. ### Reproduction Steps build clr+libs -rc Release cd src\libraries\System.Collections.Immutable\tests dotnet test -f:net462 ### Expected behavior Tests run. ``` Discovering: System.Collections.Immutable.Tests (app domain = on [no shadow copy], method display = ClassAndMethod, method display options = None) Discovered: System.Collections.Immutable.Tests (found 4008 of 4123 test cases) Starting: System.Collections.Immutable.Tests (parallel test collections = on, max threads = 16) ``` ### Actual behavior No tests run: ``` Test run for c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll (.NETFramework,Version=v4.6.2) Starting test execution, please wait... A total of 1 test files matched the specified pattern. No test is available in c:\src\dotnet\runtime\artifacts\bin\System.Collections.Immutable.Tests\Debug\net462\System.Collections.Immutable.Tests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. ``` ### Regression? I think this is a recent regression. I believe this used to work as I remember seeing test failures on framework when running with dotnet test. ### Known Workarounds Directly invoke the `test` target. > \src\dotnet\runtime\.dotnet\dotnet build /t:test -f:net462 ### Configuration _No response_ ### Other information _No response_
Author: ericstj
Assignees: -
Labels: `area-Infrastructure-libraries`, `untriaged`
Milestone: -
ViktorHofer commented 10 months ago

Someone needs to debug this. This likely also impacts VS Text Explorer which uses the VSTest runner underneath. There are settings to get more verbose logs returned by VSTest.

ViktorHofer commented 10 months ago

One more thought, that could be related to the recent xunit 2.5.x upgrade.

ViktorHofer commented 10 months ago 

TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.757, 3877444491007, vstest.console.dll, MetadataReaderExtensionsHelper: Discovering extensions inside assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.8.21101, Culture=neutral, PublicKeyToken=31bf3856ad364e35' file path 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Diagnostics.NETCore.Client.dll'
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444578798, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolving assembly.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444580691, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Searching in: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.766, 3877444582066, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Bcl.AsyncInterfaces.dll', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583023, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Extensions\Microsoft.Bcl.AsyncInterfaces.exe', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583316, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Searching in: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444583907, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Microsoft.Bcl.AsyncInterfaces.dll', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444584450, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Assembly path does not exist: 'C:\git\runtime2\.dotnet\sdk\8.0.100-rtm.23506.1\Microsoft.Bcl.AsyncInterfaces.exe', returning.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444584752, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Failed to load assembly.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444585113, vstest.console.dll, CurrentDomainAssemblyResolve: Resolving assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444585764, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolving assembly.
TpTrace Information: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586390, vstest.console.dll, AssemblyResolver.OnResolve: Microsoft.Bcl.AsyncInterfaces: Resolved from cache.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586674, vstest.console.dll, CurrentDomainAssemblyResolve: Resolving assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.767, 3877444586988, vstest.console.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Verbose: 0 : 25352, 1, 2023/10/30, 19:39:45.768, 3877444596875, vstest.console.dll, CurrentDomainAssemblyResolve: Failed to resolve assembly 'Microsoft.Bcl.AsyncInterfaces'.
TpTrace Warning: 0 : 25352, 1, 2023/10/30, 19:39:45.783, 3877444748536, vstest.console.dll, TestPluginDiscoverer: Failed to get types from assembly 'Microsoft.Diagnostics.NETCore.Client, Version=0.2.8.21101, Culture=neutral, PublicKeyToken=31bf3856ad364e35'. Error: System.Reflection.ReflectionTypeLoadException: Unable to load one or more of the requested types.
Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
   at System.Reflection.RuntimeModule.GetTypes(RuntimeModule module)
   at System.Reflection.RuntimeModule.GetTypes()
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginDiscoverer.GetTestExtensionsFromAssembly[TPluginInfo,TExtension](Assembly assembly, Dictionary`2 pluginInfos, String filePath) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginDiscoverer.cs:line 158
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.
File name: 'Microsoft.Bcl.AsyncInterfaces, Version=1.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'
---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Bcl.AsyncInterfaces, Culture=neutral, PublicKeyToken=null'. The system cannot find the file specified.
File name: 'Microsoft.Bcl.AsyncInterfaces, Culture=neutral, PublicKeyToken=null'
   at System.Reflection.RuntimeAssembly.InternalLoad(AssemblyName assemblyName, StackCrawlMark& stackMark, AssemblyLoadContext assemblyLoadContext, RuntimeAssembly requestingAssembly, Boolean throwOnFileNotFound)
   at System.Reflection.Assembly.Load(AssemblyName assemblyRef)
   at Microsoft.VisualStudio.TestPlatform.Common.ExtensionFramework.TestPluginCache.CurrentDomainAssemblyResolve(Object sender, AssemblyResolveEventArgs args) in /_/src/Microsoft.TestPlatform.Common/ExtensionFramework/TestPluginCache.cs:line 513
   at System.Runtime.Loader.AssemblyLoadContext.GetFirstResolvedAssemblyFromResolvingEvent(AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingEvent(AssemblyName assemblyName)
   at System.Runtime.Loader.AssemblyLoadContext.ResolveUsingResolvingEvent(IntPtr gchManagedAssemblyLoadContext, AssemblyName assemblyName)

--diag:log.txt

ViktorHofer commented 10 months ago

Filed https://github.com/xunit/visualstudio.xunit/issues/390

bradwilson commented 10 months ago

This is not what you think it is.

image

bradwilson commented 10 months ago

Are you're linking against xunit.runner.utility for some reason? If so, then you're frozen to using whatever version of xunit.runner.utility the VS adapter uses (because VSTest demands all dependencies be copied into the output folder). 99%+ of users don't do this so this isn't a general problem.

bradwilson commented 10 months ago

The only reason to link against xunit.runner.utility is because you're creating your own runner. Tests should never link against this unless they're forced to because they're testing the runner you're writing.

bradwilson commented 10 months ago

For a bit of architectural background...

xUnit.net splits what it does into two separate sections which do not directly interact except via very strict contracts. There are "runner" and "execution" halves of the system. In v1/v2 when using .NET Framework, these two halves can be optionally separated by an app domain boundary (and in v2 .NET Core, they live in the same process). In v3 against all target platforms, they're separated by a process boundary.

By keeping this strong separation and frozen contracts, we have both backward compatibility (newer runners can run older tests) and major forward compatibility (older v2 runners can run newer v2 tests). In point of fact, I just was able to successfully run tests linked against core framework 2.5.3 with the VSTest adapter 2.0.0, which is 8.5 years old and targeted net20, wpa81, and win8. Obviously .NET Core didn't even exist back then, so it wouldn't be able to run a platform it doesn't know about, but .NET Framework is not a problem.

If your tests link against xunit.runner.utility.*, you're bringing the runner half of things where it normally doesn't live or belong. This is unavoidable if you're writing your own runner, and want to test that runner (we obviously do this, but it should be very very rare). Normally this wouldn't be a problem except when you're using VSTest, because there is a requirement placed by VSTest that we copy the VSTest adapter into your output folder, which of course means we also have to copy the runner utility libraries (xunit.runner.utility.* and xunit.runner.reporters.*). This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you're in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you're using.

In addition, there is no guarantee that any particular version of xunit.runner.visualstudio will use any particular version of xunit.runner.utility. We shipped 2.5.3 of the former at the same time as 2.5.2 of the latter, but even simultaneous shipment isn't guaranteed (and in fact they lived very separate lives back when @clairernovotny owned the VSTest adapter).

When I shipped core framework 2.5.3, it had no runner utility changes that would've necessitated bringing in a new version for the VSTest adapter, so I didn't bother to ship a new one. It appears that when I ship the next core framework drop (which I'm planning to number 2.6.0), this will still be true, so again I'm not likely to ship another VSTest adapter version.

ViktorHofer commented 10 months ago

Wow, that's quite a detailed comment. Thank you for spending your time on writing this up. I just reproduced this locally and I don't see our test assemblies or test helper assemblies linking against any of the xunit.runner.* assemblies.

What I see is that because of us wanting to support both xunit.console and VSTest, we bring in xunit.runner.utility.net452.dll from the xunit.console and explicitly exclude the one from xunit.runner.visualstudio which results in that version collision.

This is the only scenario where we end up with potential collisions between versions of the runner utility libraries, so when you're in this scenario, you must very explicitly remain frozen to use the same version of the runner utility libraries as the VSTest adapter is that you're using.

Exactly that.

This begs the question why we ever chose a design that made us copy the xunit.console runner into the test app's output directory. While VSTest requires that, xunit.console shouldn't.

bradwilson commented 10 months ago

Yeah, that operation makes no sense to me. You were basically just waiting to explode. 😁

Each runner has to bring along their own copy of xunit.runner.utility.* that they're linked against. It's just the VSTest adapter which insists on copying it into your output folder; all the other runners just live elsewhere and are invoked by hand, so the assemblies are never mixed together.

bradwilson commented 10 months ago

To be even more explicit: xunit.console would never use xunit.runner.utility.* from the test's folder. It would need to live along side xunit.console.exe so that .NET Framework could do the appropriate implicit loading. If you're copying the contents of xunit.console's NuGet package into your test output folder, you should stop doing that. It's unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages or some equivalent).

ViktorHofer commented 10 months ago

you should stop doing that. It's unnecessary and counterproductive. Just run it from wherever it lives (in ~/.nuget/packages or some equivalent).

Just to share some context. We invoke our tests in CI on different machines than the build agents. We use Helix for that, which is a distributed machine testing orchestration infrastructure. Because of that, we can't rely on the package cache. Instead we should have just sent that dependency as an additional "payload" (dependency) to that helix test agent.

Unfortunately our infrastructure here has become quite complex with so many different app models and ways of testing that this change will probably require some time.

But for what is worth, this is indeed our own fault. Thanks for help diagnosing the issue :)

bradwilson commented 10 months ago

Good luck! 😄

ericstj commented 8 months ago

@ViktorHofer did you have a suggestion for what needs to happen to fix this?

I gather that by copying xunit runner binaries over to the output directory this is somehow causing the test-discoverer to fail before it gets a chance to examine our test assembly, is that correct?

ViktorHofer commented 8 months ago

The gist (IIRC) is to not copy xunit.console's assemblies into the output directory and just invoke it from the package cache (as intended). Only copy it into the output directory when archiving the test assemblies for helix submissions (property: /p:ArchiveTests=true).

xunit.runner.visualstudio is designed to be copied into the app output directory but xunit.console isn't. So avoiding the latter makes sense on a local machine.

bradwilson commented 8 months ago

xunit.runner.visualstudio is designed to be copied into the app output directory

That makes it sound like we do something special here. We don't. The copy is an unfortunate requirement from the VSTest team; if we don't copy the DLLs into the output directory, then whoever runs dotnet test (or equivalent) needs to specify the location of the test adapter path on disk via an additional command line argument.

bradwilson commented 8 months ago

If it helps, we could add an MSBuild property that we look for to prevent the file copies. They would need to stay on by default but if you know what you're doing, skipping the copies could make sense.

ViktorHofer commented 8 months ago

I think this really is an issue on our side and needs a fix in our test infrastructure. The xunit.console copying is just wrong and also in-efficient. I should have done this differently in the past.

ericstj commented 8 months ago

Yeah, other repos don't copy and bundle the runner package as a correlation payload. It's not so hard to make that work, but it does require changes in a few places.

One thing that was odd to me is that we're saying that our binaries are self-consistent and would work if we merely delete files that aren't needed. That tells me we might also have a bug / design-change ask for the dotnet test scenario to not halt when it encounters binding problems in other assemblies in the output directory.

ViktorHofer commented 8 months ago

It's not so hard to make that work, but it does require changes in a few places.

I was suggesting something simpler for a quick fix. Keep the xunit.console in the app output directory but only when ArchiveTests=true. Otherwise (locally) use the xunit.console from the nuget cache.

ericstj commented 7 months ago

@akoeplinger found other places that still rely on xunit.console in the output directory.

it looks like https://github.com/dotnet/runtime/pull/96826 is breaking a couple things

The latter seems more problematic since we document /t:Test in a couple places and I know many people are still in the habit of using it vs dotnet test and in some cases you don't want the vstest runner

It seems we have some more places to root out where we expect the runner in the output directory - those need to either be updated to pick up the runner from the package, or opt-in to copying.

ViktorHofer commented 7 months ago

The latter seems more problematic since we document /t:Test in a couple places and I know many people are still in the habit of using it vs dotnet test and in some cases you don't want the vstest runner

dotnet build /t:Test should still work with https://github.com/dotnet/runtime/pull/96826. What are we missing here?

ViktorHofer commented 1 month ago

Moving this back into 9.0.0 as this makes debugging .NET Framework tests with dotnet test harder. We can move it back to 10.0.0 if we don't get to it though.