NUnit.ConsoleRunner.NetCore executes test library under wrong runtime version

[MichalMucek]

I'm trying to execute a test library using NUnit Console 3.16.3 (NuGet, .NET Tool) on Windows Server 2019 Datacenter (10.0.17763, x64). There are multiple .NET SDK versions installed on the machine - 6.0.202, 7.0.100, 7.0.309, 8.0.100.

In use are NUnit 3.13.1, NUnit3TestAdapter 4.4.2, and SpecFlow.NUnit 3.9.74.

The TargetFramework of the library is either net7.0-windows10.0.19041.0 or net8.0-windows10.0.19041.0. Each time I execute one of the libraries, .NET 6.0.23 (greatest/latest of the other available 6.0.x) is used for runtime, which results in errors of missing assemblies for the targeted framework or the selected runtime. In the case of net7.0-windows10.0.19041.0, it can reach the point of finding the tests. It is unable to do even that for net8.0-windows10.0.19041.0.

The test library executes with no issues when I set up an environment with .NET 7 or 8 only (or when I make a few tweaks in the appropriate keys of the Windows Registry). Using global.json with a specific .NET SDK version doesn't work.

InternalTrace.9572.Org.App.Specs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null (.NET 7).log InternalTrace.9572 (.NET 7).log Console output for net7.0-windows10.0.19041.0:

dotnet tool run dotnet-nunit .\Org.App.Specs.dll --trace=Debug
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Thursday, January 11, 2024 11:44:21 AM

Runtime Environment
   OS Version: Microsoft Windows 10.0.17763
  Runtime: .NET 6.0.23

Test Files
    .\Org.App.Specs.dll

Errors, Failures and Warnings

1) SetUp Error : Org_App_Specs_NUnitAssemblyHooks
System.IO.FileNotFoundException : Could not load file or assembly 'System.Collections.Specialized, Version=7.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.
   at System.Configuration.ConfigurationElement..ctor()
   at System.Configuration.ConfigurationElementCollection..ctor()
   at BoDi.ContainerRegistrationCollection..ctor()
   at TechTalk.SpecFlow.Configuration.ConfigurationLoader.GetDefault()
   at TechTalk.SpecFlow.Infrastructure.ContainerBuilder.CreateGlobalContainer(Assembly testAssembly, IRuntimeConfigurationProvider configurationProvider)
   at TechTalk.SpecFlow.TestRunnerManager.CreateTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder)
   at TechTalk.SpecFlow.TestRunnerManager.GetTestRunnerManager(Assembly testAssembly, IContainerBuilder containerBuilder, Boolean createIfMissing)
   at TechTalk.SpecFlow.TestRunnerManager.OnTestRunStart(Assembly testAssembly, IContainerBuilder containerBuilder)
   at Org_App_Specs_NUnitAssemblyHooks.AssemblyInitialize() in C:\ApplicationRepo\Tests\App\Org.App.Specs\obj\Release\net7.0-windows10.0.19041.0\NUnit.AssemblyHooks.cs:line 20

Run Settings
    DisposeRunners: True
    InternalTraceLevel: Debug
    WorkDirectory: C:\net7.0-windows10.0.19041.0
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 678, Passed: 0, Failed: 678, Warnings: 0, Inconclusive: 0, Skipped: 0
    Failed Tests - Failures: 0, Errors: 678, Invalid: 0
  Start time: 2024-01-11 11:44:21Z
    End time: 2024-01-11 11:44:22Z
    Duration: 0.916 seconds

Results (nunit3) saved as TestResult.xml

InternalTrace.3312.Org.App.Specs, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null (.NET 8).log InternalTrace.3312 (.NET 8).log Console output for net8.0-windows10.0.19041.0:

dotnet tool run dotnet-nunit .\Org.App.Specs.dll --trace=Debug
NUnit Console 3.16.3 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Thursday, January 11, 2024 11:44:51 AM

Runtime Environment
   OS Version: Microsoft Windows 10.0.17763
  Runtime: .NET 6.0.23

Test Files
    .\Org.App.Specs.dll

Errors, Failures and Warnings

1) Invalid : C:/net8.0-windows10.0.19041.0/Org.App.Specs.dll
Could not load type 'System.Runtime.CompilerServices.NullableContextAttribute' from assembly 'System.Runtime, Version=6.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

Run Settings
    DisposeRunners: True
    InternalTraceLevel: Debug
    WorkDirectory: C:\net8.0-windows10.0.19041.0
    NumberOfTestWorkers: 8

Test Run Summary
  Overall result: Failed
  Test Count: 0, Passed: 0, Failed: 0, Warnings: 0, Inconclusive: 0, Skipped: 0
  Start time: 2024-01-11 11:44:51Z
    End time: 2024-01-11 11:44:52Z
    Duration: 0.327 seconds

Results (nunit3) saved as TestResult.xml

[stevenaw]

Hi @MichalMucek Unfortunately there are a few issues with the 3.16 version of the console runner. Since you have .NET8 installed could you please see if the prerelease of 3.17 works for you? That version should include .NET8 support among other fixes. https://github.com/nunit/nunit-console/releases/tag/V3.17.0-dev.3

Note that there is still a known issue related to projects requiring certain particular runtimes, with a workaround defined in https://github.com/nunit/nunit-console/issues/1383. If you hit the issue the workaround should work if your tests do not require a non-Windows OS.

[MichalMucek]

@stevenaw Thanks for responding. The release version of 3.17.0 produces the same issue.

I submitted #1381 not only with the intention of making the NUnit console work in environments where the .NET SDK is not installed, but also with the hope that it will be a workaround for this issue.

[OsirisTerje]

@MichalMucek Have you worked through the comments in the #1374 and see if any of those might apply to you?

[MichalMucek]

@OsirisTerje Thanks for the tip. I've tried tweaking the nunit3-console.runtimeconfig.json file which was located at .../.nuget/packages/nunit.consolerunner.netcore/3.17.0/tools/net6.0/any. Changing runtimeOptions.framework.version from 6.0.0 to 8.0.0 helped.

[MichalMucek]

Today, I installed NUnit.ConsoleRunner.NetCore 3.17.0 (.NET tool) on some other environment. There was a warning message (colored dark yellow), saying Format version is missing. This tool may not be supported in this SDK version. Contact the author of the tool. The used .NET SDK version was 8.0.200.

File nunit3-console.runtimeconfig.json was still pointing at framework version 6.0.0.

[OsirisTerje]

@MichalMucek I can confirm this, so should be fixed in https://github.com/nunit/nunit-console/pull/1396

[CharliePoole]

@OsirisTerje Is this a won't fix for version 3?

NUnit.ConsoleRunner.NetCore, executed directly or as a dotnet tool, only runs tests in-process. Since it targets .NET 6.0, that's how it runs them. Of course, the user may force it to run under .NET 7.0 or 8.0, but the code is still .NET 6.0 code.

I have some thoughts about giving the NetCore runner all the same capabilities as the standard console runner. That, however is a fairly big piece of work and seems more like a version 4 thing to me. Essentially, we would eliminate the distinction between the two runners and they would both do the same thing. In the distant future, we might even phase out the netfx runner, but I wouldn't hold my breath.

For the moment, I would suggest to @MichalMucek to do one of two things.

1. Use dotnet test
2. Use the standard runner, starting with the upcoming 3.18.0 release

Thoughts on this? Should we enhance the dotnet runner for 3.x?

[kogoel]

The latest version 3.18.2 of NUnit.ConsoleRunner.NetCore has the same behavior and taking Runtime as .net 6.0.5 even when .Net 8.0.8 is installed, with below error: Unable to load one or more of the requested types. Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

[MichalMucek]

Another workaround for the issue is not having any .NET SDK installed...

We handle .NET SDK on our test environment with the .NET SDK Support Jenkins plugin. It allows us to use different .NET SDK versions without installing it. We have a test environment with 28 machines and it would be cumbersome to edit manually nunit3-console.runtimeconfig.json with every new version of the NUnit Console.

An alternative is to use a zipped (with binaries) version and perhaps some temporary additions to environment variables.

[bouchraRekhadda]

The latest version 3.18.2 of NUnit.ConsoleRunner.NetCore has the same behavior and taking Runtime as .net 6.0.5 even when .Net 8.0.8 is installed, with below error: Unable to load one or more of the requested types. Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

@kogoel Is your DLL self-contained ? because that typically happens when you are trying to do dotnet nunit3-console.dll --explore on a a self-contained library (reproducible with 3.18.1)

[CharliePoole]

@OsirisTerje @MichalMucek @bouchraRekhadda @kogoel

This issue has had no attention since my comment three months ago asking for some overall direction for NUnit.ConsoleRunner.NetCore. I flagged it as needing discussion, which has not yet happened.

It was probably a mistake on my part to ask for that general discussion on this specific issue. I'm going to pull together what's done and what remains undone, along with the much greater part of potential functionality, which is undecided, and create a new issue to summarize all of it. I'll link to it here when I create it.

Meanwhile, I would like to refocus this issue on the original problem reported by @MichalMucek.

[CharliePoole]

@MichalMucek Do you have a simple repro I can use to work this issue? Preferably one without third-party references?

[CharliePoole]

@MichalMucek Also, the command-line used to execute your problem test scenario.

[OsirisTerje]

@CharliePoole Sorry for not coming back on your comment, but I actually don't know. The console runners are not really following SemVer, so whether it should be 3 or 4 - I don't know. I would say that IF there are breaking changes, then it is a 4. If it is just enhancements, and everything works as before for those people using it, well, then it is a 3. But, if we feel the change is a big one, we could still give it a 4, to indicate this is a big change. In this case, it may be a lot of change to the code, as you indicate, but for the user - not so much. What I think is important, is what you have been doing lately, fixing issue by issue, and then unblocking a lot of stuff. To me it looks like the proposed changes here are the same, unblocking stuff. If that can be done in small steps, so that we dont have long living feature branches - which may go stale, that would be awesome. And yes, it deserves a new issue.

[CharliePoole]

@OsirisTerje My question is more fundamental than version three vs four. Once we have a specific feature in mind, it's easy to figure out if it's breaking or if it's scope is so great that we prefer to wait for V4. It would be great to have some discussion about that when we have V3 a bit cleaner.

What I don't actually know is the intended feature set for the NetCore runner. One way to do that may be to look at the limitations and then decide if it's desirable to remove them or if they are reasonable for the intended use of this runner. That's what I'll write something more about in a separate issue. Soon. :-)

BTW I recognize that you may be in the same boat as I am, i.e. not a routine user of the NetCore runner or the dotnet tool that is based on it. In that case, we need to find some people who are. @NUnit team? Users?

[OsirisTerje]

When I am using the console runner, I am actually using the netcore one, as I install it as a dotnet tool. That makes it very convenient to use. I rarely use any of the other console apps.

But, for the use I have been doing, I have not been bothered by the limitations there. I noticed them, but that was all. That said, it would be nice to get it more aligned with its more mature siblings, but then I am not the "routine user".

[MichalMucek]

@MichalMucek Do you have a simple repro I can use to work this issue? Preferably one without third-party references?

@CharliePoole No, not at the moment. I'll try to work out something but please don't expect it soon.

[bouchraRekhadda]

NUnit.ConsoleRunner.NetCore, executed directly or as a dotnet tool, only runs tests in-process. Since it targets .NET 6.0, that's how it runs them. Of course, the user may force it to run under .NET 7.0 or 8.0, but the code is still .NET 6.0 code.

@CharliePoole how can we force it to load other runtime versions? also would it be feasible for the Nunit.ConsoleRunner.NetCore to target .NET 8 directly ?

[CharliePoole]

@bouchraRekhadda

how can we force it to load other runtime versions?

Some people report success changing the delivered .runtimeconfig.json file.

also would it be feasible for the Nunit.ConsoleRunner.NetCore to target .NET 8 directly ?

Yes, I plan to make a build available for testing. It will need to be tested both by some folks targeting .NET 6.0 and earlier releases as well as those who use .NET 8.0. Ideally, we can find some people to test it both in the form of a tool and by running it directly.

[bouchraRekhadda]

@CharliePoole

Yes, I plan to make a build available for testing. It will need to be tested both by some folks targeting .NET 6.0 and earlier releases as well as those who use .NET 8.0. Ideally, we can find some people to test it both in the form of a tool and by running it directly.

If the nupkg will be available for me to download I can test it (we are currently targeting .NET 6 and working to upgrade to .NET 8 soon)

[CharliePoole]

Excellent. I'll post a link here when it's ready.

[bouchraRekhadda]

Hello @CharliePoole do you know when will this package be available to test it ?

Regards

[CharliePoole]

The release won't be until 3.19 (I'm now working on 3.18.3) but I plan to have a build for testing available in a few days.

[CharliePoole]

@bouchraRekhadda There is a .NET 8.0 console runner avaialble on the myget feed. It's NUnit.ConsoleRunner.Net80 version ~3.18.3-dev00006~ UPDATED: 3.18.3-dev00009. This is an experimental build so the name may not be permanent. I expect to have an equivalent runner in 3.19 either as an update to the netcore runner or an alternative.

[CharliePoole]

See #1493, which I'm hoping will resolve this issue as well.

[CharliePoole]

@MichalMucek Can you try the latest dev build from our myget feed? It includes NUnit.ConsoleRunner.Net80, which I believe will solve your problem. It's an experimental build, so only available as a nuget package. The tool name is nunit-net80.

[kogoel]

@CharliePoole : Can you please confirm the nuget package in which this bug is fixed is https://www.nuget.org/packages/NUnit.ConsoleRunner.Net80 Thanks Kohima

[CharliePoole]

@kogoel

The issue is still outstanding but the 3.18.3 release contains an experimental .NET 8.0 runner, NUnit.ConsoleRunner.Net80, available on NuGet or on the GitHub release page.

https://www.nuget.org/packages/NUnit.ConsoleRunner.Net80

See also the discussions under #1492 and #1493

[kogoel]

@kogoel

The issue is still outstanding but the 3.18.3 release contains an experimental .NET 8.0 runner, NUnit.ConsoleRunner.Net80, available on NuGet or on the GitHub release page.

https://www.nuget.org/packages/NUnit.ConsoleRunner.Net80

See also the discussions under #1492 and #1493

Yes tested with the mentioned Nuget package received the error similar to earlier but now the error says, if it helps. Could not load file or assembly 'System.Security.Cryptography, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. while earlier it was Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified. Thanks Kohima