search

nunit / nunit-console

NUnit Console runner and test engine
MIT License
215 stars 152 forks source link

Improvements to the NETCORE console runner #1101

Open CharliePoole opened 2 years ago

CharliePoole commented 2 years ago

I'm writing this to replace issue #858. That issue has become cluttered with a lot of confusing comments... many of them my own. :-( The confusion largely arises because it wasn't always clear whether we were talking about the NETFX runner or the NETCORE runner... both of which have the same name!

In closing #858 I summarized the problem in a comment, which i'll repeat here...

  1. The NETCORE console runner is (unfortunately) called nunit3-console just like the NETFX runner, but it has entirely different capabilities.

  2. The NETCORE runner does not launch agent processes to isolate tests but runs them in-process.

  3. The NETCORE runner has some support for engine extensions, but of course only for extensions, which target .NET Core or .NET Standard. I only know of one such extension... our V2 Result Writer. All our other extensions are NETFX only.

  4. The NETCORE runner itself targets .NET Core 3.1 and is set up to allow rollover to new major versions if 3.1 is not installed. That means the tests it runs have to target .NET Core 3.1 or lower.

  5. Since there is no VS project loader extension compatible with .NET Core, the NETCORE runner can only run assemblies entered at the command-line.

(Everyone... please correct me if any of the above seems wrong!)

I'd like to use this issue to restart the discussion about our NETCORE runner (or runners) and define more clearly where we want to go in coming releases. Once we have made our decisions, this Epic issue will be used to gather all the sub-issues we create to implement them.

CharliePoole commented 2 years ago

Here is a list of topics I think we need to discuss and make decisions on. I have tried to think very broadly about this and not limit myself to what we are already doing. Feel free to do the same as you suggest additions or changes.

  1. Is there only one NETCORE runner, do we need two of them? I'm thinking we may have a need for a .NET Core build of the full console runner in the future. After all, .NET FX won't be around forever. OTOH, the runner that is used as a dotnet global or local tool might be expected (by users) to have different capabilities.
  2. Should the NETCORE runner - or at least the one used as a tool - have a different name?
  3. What capabilities should that runner have? Should it launch processes? Support extensions? Handle projects in addition to assemblies? Solutions?
  4. If the runner used as a tool doesn't launch processes, shouldn't it then target the latest release of .NET?
  5. Again... if we are not launching processes... do we need to use the engine at all? Would it be simpler to call the framework directly as nunitlite does?
CharliePoole commented 2 years ago

@nunit/engine-team Ideas and comments? This seems like a really key decision to make.

jpactor commented 2 years ago

Will there be support for net6.0 now that it's released (three months ago!)? net5.0 won't be LTS, only net6.0, and MS has made it clear that net6.0+ is "the future."

CharliePoole commented 2 years ago

I absolutely agree. But we need to go a bit further and specify what "support" actually means. One possibility is that we simply build the runner for .NET 6.0 rather than .NET Core 3.1. That would of course mean that your pre-6.0 tests would be running under .NET 6.0. Is that a drawback for some folks? I don't know... which is basically the reason that this item needs discussion. :-)

jpactor commented 2 years ago

I noticed in the runner package (link: https://nuget.info/packages/NUnit.ConsoleRunner/3.14.0) that there appears to be support for netcoreapp3.1 and net5.0, so maybe adding a net6.0 target framework is enough?

Not sure it has to be a "rather than" situation.

CharliePoole commented 2 years ago

In the case of NUnit.ConsoleRunner, you're correct. In fact, there's already an issue to add a .NET 6.0 agent.

However, this issue is about the NETCORE console runner, NUnit.ConsoleRunner.NetCore. That runner doesn't have agents for running out of process. Your tests run directly using the target framework for which the runner is built. The NETCORE runner is the one you are using if you type dotnet nunit at the command line. The standard NUnit runner is the one you get by typing nunit3-console.

One of the things I want to do is find a new name for this runner package to avoid confusion... hence point 2. :-)

So... which runner are you using?

jpactor commented 2 years ago

The latter (NUnit.ConsoleRunner). Sorry for the confusion! And I see the net6.0 agent is coming in 3.15!

CharliePoole commented 2 years ago

Yes, it is... I delayed it because it requires changing our build to use VS2022.

mikkelbu commented 2 years ago

First a disclaimer, I've don't think I've used the NETCORE console runner nor any other .Net tools except for the default ones (at least as I know), so my knowledge in this area is very limited.

  1. We probably need two, but this is just a guess. I'm unsure if there are any constraints/restricts on .Net tools (e.g. that they cannot spawn processes) or if they can do "anything"
  2. I think so. There are already some confusion about the different runners due to the naming
  3. Existing users probably expect it to have the same features as the old runners, but I would guess that we only need to support a limited subset of the extensions, i.e. NUnitProjectLoader and VSProjectLoader (or NETCORE variants thereof). I don't think the NUnitV2.... extensions makes sense in this setting and the same for the TeamCityEventListener.
  4. Sounds reasonable
  5. I don't know. My knowledge of the nunit-console code is limited, so I don't know if there are any features in the engine that are not in nunitlite (or vice versa)
CharliePoole commented 2 years ago

Thanks for the comments @mikkelbu

For 3.x I'll keep the existing structure (one project, two builds) and capabilities. The five points above will be the basis for some breaking changes as well as some improvements in the tool for 4.0.

For 3.15

For 4.0

MJB222398 commented 6 months ago

I am trying to run the console runner as a dotnet tool (think this is the "NETCORE" you refer to) on a project targeting .NET 8. The runner fails:

NUnit Console 3.17.0+685c5b542b5e9ba632c905f0bd514a773d9758af (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
24 April 2024 09:56:12

Runtime Environment
   OS Version: Microsoft Windows 10.0.19045
  Runtime: .NET 6.0.29

Test Files
    Gubbins/bin/Release/net8.0/Gubbins.dll

Errors, Failures and Warnings

1) Invalid : C:\Users\Me\source\repos\Playground\Gubbins\bin\Release\net8.0\Gubbins.dll
Unable to load one or more of the requested types.
Could not load file or assembly 'System.Runtime, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

I can't see any actual resolution being proposed and implemented here. Why is that? .NET 8 has been out for a long time now, why does this not work? Is it possible for me to make this work or does it require changes on your side?

ilCosmico commented 6 months ago

@MJB222398, please take a look at issue #1383. It seems that there is currently no one available to address this issue. Any insights or contributions would be greatly appreciated.