v4 Console/Engine Platform support

NUnit has a long history of supporting legacy platforms. For v4, I wanted to think about how we can continue to do that, whilst reducing the burden on the development team. This post is intended to start that discussion.

Some high level thoughts:

I'd like to see all the code in the main Engine/Console project targeting modern platforms.
I'd like us to be able to continue to run tests on legacy platforms in an arms-length manner.
I'd like to do whatever possible to reduce platform lock-in, and allow us to be more flexible in our commitment to supporting platforms.

Here's how I was thinking we could address that, component-by-component:

The Console

The NUnit Console has historically been restricted to targeting .NET 2.0, as we've committed to being able to run tests on platforms with only .NET 2.0 installed.

.NET Core brought about self-contained executables, which allow us to run on any platform, with no dependency on any runtimes being installed. To me, a truly portable test runner exactly where we want to be. This would then mean that in future, we can upgrade the console's platform as we see fit, as opposed to being restricted to targeting the lowest-supported platform.

The Engine Agents

It is necessary to have an engine agent for every platform we wish to run tests on. I would like us to look at making it possible to add agents via the engine's extensibility functionality, so that users can set up the NUnit Engine can support a range of platforms, and the Engine Team aren't required to support all of these.

I'd like us to look at moving the CLR v2 agent out to an extension, and removing it from the main codebase, as we did with e.g. NUnit 2 functionality at the start of NUnit 3. .NET 2.0 is nearly 20 years old, and Microsoft's support for it ended just under 10 years ago. Doing this would mean there was a viable method for users to continue running tests on CLR v2, but it was not part of the main supported codebase.

.NET Framework and .NET Core agents should of course part of the main distribution - I hope the latter will be added during the 3.x series.

The Engine

The main engine assembly appears to be the main place where we will still be restricted on the platforms we support, as the platform the engine targets of course affects runners that rely on it.

I would like to see two ideas considered:

Removing in-process test running entirely. This would provide complete separation between the target platform of the engine, and the target platform of tests we support running.
Aiming for a single-build, which can be utilised by runners on all operating systems. Right now .NET Standard 2.0 may be the most suitable option, but that picture may well change over the time it takes to develop v4 and as .NET 5 is released - and I think it would be sensible to review the exact platform further down the line.

The second change would mean that we would be dropping support for test runners targeting .NET Framework 2.0-4.6.0. My view is that those wishing to build a test runner for these platforms should instead be looking at self-contained deployment, as we would be doing ourselves for the Console. I don't think we should hold back Engine development to legacy platforms, now an improved solution for building test runners that run on older platforms exists.

Engine API assembly

Funny one - I currently see the API assembly as serving two distinct purposes.

The API for runners wishing to reference the engine
The assembly to be referenced by engine extensions

Based on the fact that driver extensions currently need to be loaded by agents, there is a potential need for extensions to be able to target legacy platforms, and thus the API assembly as well. This is a restriction that's only necessary for "purpose 2" however - so I wonder if there's value in splitting the API assembly? This would give us e.g.:

nunit.engine.api.dll: Contains all the code in the current API assembly, targets the same platform(s) as nunit.engine.dll
nunit.extensions.api.dll: A new, super-slim assembly, which contains only the interfaces required by extension developers (IProjectLoader, IResultWriter etc.). This assembly could continue to target the lowest platform we see fit - potentially .NET 2.0.

Mono

Final little thorn here - as long as I've worked on NUnit, we've lacked willing contributors to invest in our Mono-specific functionality. My ideal would be for a member of the Mono community to step up and commit to providing support for mono, however no such person has materialised over the course of NUnit 3. On this basis, in order to only commit to what the team can continue to support, I'd like to review our level of Mono support, including:

Removal of our Mono specific functionality. I think this is currently only installed framework discovery and targeting.
A commitment to continue running our tests on mono, as long as our CI systems reasonably support this scenario. However, a disclaimer that we may chose to exclude new features from running on mono, if the volume of additional work to support the platform is more than a contributor is prepared to undertake.

Side note, we currently have a method to invoke running test on mono from the .NET Framework console. I have no idea if anyone uses it, or even if it works - but I wonder if that could also be moved to an extension, as per the .NET 2.0 agent above.

This ended up a little longer than I expected - I've laid out a lot of ideas, but very much invite everyone's thoughts and feedback on them. I'm intending this to start the discussion, rather than be a design document quite yet. 🙂

A thought I keep having about all of our projects is that in ~~2009~~, NUnit only supported back to .NET Framework 2.0. It didn't support .NET Framework 1.1 which was six years in the past at that point. The console still supports .NET Framework 2.0 which is now 15 years in the past. What would it look like if we had kept a five-year max the whole way? .NET Framework 4.6 would be the oldest thing our projects would support.

Correction: I couldn't find this when I looked at the initial Git commit in this repo, but NUnit actually supported .NET Framework 1.1 up to 2012 in the main package and up to 2014 through a separate package. Thanks for the info, @CharliePoole!

That's a very reasonable point!

With the console/engine, I think it's a little easier to allow test running on older platforms. "Agent extensions" are something we want to tackle anyway, which I think gives us a nice way to allow others to run CLR v2 tests, without us having to support them. (I didn't include it above, but I'd like to see us create and publish the CLR v2 agent extension, then archive the repository shortly after - unless there was a willing maintainer to take it on.)

That just leaves the nunit.extensions.api.dll - which I expect to be such a small collection of very simple interfaces, there doesn't really seem to be much reason to not support the older platforms.

For v4 of the framework however, there would be some bigger decisions to be made!

Now we're a little closer, let's try and put some platforms to these assemblies. Here's my thoughts, to get us started:

Console Two targets: .NET 4.x (latest) and .NET 5.

Engine Latest target that can be referenced by both latest .NET Framework and .NET Core. I believe that's currently .NET Standard 2.0.

Agents Agents to be included for CLR v4. Target .NET Framework 4.0. (No significant advantage to drop 4.0 support?) Agents to be included in future for .NET 5.0, and possibly .NET Core 3.1 (Would be a new feature, so not necessary for v4.0. Dependent on #266) Agents for CLR v2 to be available via an extension and Pluggable agents. (N.B. This would require a pluggable agents feature to make this breaking change) Agent for running on mono to be available via an extension/Pluggable agents

Engine Core Required by packaged agents and engine, therefore to target net40/netstandard20. Separate build of this will need to be exported/archived along with clr v2 agent extension

nunit.engine.api netstandard20, as per main nunit.engine assembly

Extension API assemblies (Structure/naming TBD)

API's for extensions loaded by main engine can target netstandard20.
Driver extensions are loaded by the agent, so API assembly needs to target lowest runtime that an agent may wish to target. Believe that's currently best to be net20/netstandard10. This assembly should be very simple, don't believe there's any need to target a higher version.
Any shared extension API assembly will need to target lowest API version too. Hopefully we won't need one of these.

I think this is one of the major breaking change issues to resolve for v4, and has a number of dependencies. Thoughts from everyone welcome.

@ChrisMaddock

Thoughts on your recap of platform support...

Console

I think I agree with that. WRT .NET Framework, I'm not sure whether you're saying the user has to have 4.8 installed in order to use the console or that we will run under whatever 4.x they have installed. The latter would mean the console targets 4.0, which makes a bit more sense to me.

Engine

No 4.0 Engine? Then that does imply the user machine must have 4.6.2 or higher installed. Seems a bit restrictive although it would force us to get the .NET Standard engine working fully.

I thought you already had a .NET Core 3.1 engine build? Did that get removed?

Agents

I agree on the list of builtin vs pluggable agents. I also agree that this is breaking if we remove the builtin 2.0 agent before we have a pluggable agent. So I see two alternatives: (1) keep the 2.0 agent for 4.0 and replace it with a pluggable agent in 4.1 (I wouldn't call this breaking) or (2) replace it in 4.0.

I think the second approach is nicer, but more work. There's a third approach, of course, where we do it before 4.0 but I think we already agreed that doing 4.0 rapidly would help us in other ways.

Engine Core

This is where I'm running into issues with my pluggable .NET 2.0 agent. I want to experiment with reducing the dependency of agents on parts of the .NET Core. At one time I talked about having the engine, engine.core and something like agent.core. That may be the way to go. I'll do my experimenting on the side or in the GUI for now.

Thanks for reading through, Charlie.

I think one thing I hadn't considered is that, with .NET 4.x being an in place upgrade, if we require net462 to run the console, we're essentially preventing running tests of net40/net452 etc. That's a pain, as it means we require multiple builds of the main engine, which I was hoping to avoid - but I think is perhaps necessary to support testing on the platforms we want to... 🙁

On the specific version of .NET 4.x, I'm reluctant to target a "new" project to an unsupported framework, especially one for which support ended over 5 years ago like net40. Perhaps we should target .NET 4.5.2, as the latest supported net4x? The framework no longer supports net40, so users who need to test on net40 will be restricted to old versions of the framework anyway.

Console I think I agree with that. WRT .NET Framework, I'm not sure whether you're saying the user has to have 4.8 installed in order to use the console or that we will run under whatever 4.x they have installed. The latter would mean the console targets 4.0, which makes a bit more sense to me.

I had meant net48, but given the above, net452?

Engine No 4.0 Engine? Then that does imply the user machine must have 4.6.2 or higher installed. Seems a bit restrictive although it would force us to get the .NET Standard engine working fully. I thought you already had a .NET Core 3.1 engine build? Did that get removed?

Yes, as per comments above, I'd hoped we could have a single engine build - but perhaps it's a bit soon. net452/netstandard2.1?

netcoreapp31 - IIRC this is specifically to allow use of the AssemblyLoadContext functionality required by the Net31CoreDriver. That functionality should move out to the agent once we have .NET Core agents - so a different engine build shouldn't be required anymore.

Agents I agree on the list of builtin vs pluggable agents. I also agree that this is breaking if we remove the builtin 2.0 agent before we have a pluggable agent. So I see two alternatives: (1) keep the 2.0 agent for 4.0 and replace it with a pluggable agent in 4.1 (I wouldn't call this breaking) or (2) replace it in 4.0.

I would very much like us to be able to option 2. I personally think this is one feature worth holding 4.0 for.

Engine Core This is where I'm running into issues with my pluggable .NET 2.0 agent. I want to experiment with reducing the dependency of agents on parts of the .NET Core. At one time I talked about having the engine, engine.core and something like agent.core. That may be the way to go. I'll do my experimenting on the side or in the GUI for now.

Yes - engine core should ideally be as minimal as possible! It's something I've looked at, and there's various low hanging fruit to move all the extension loading and framework services up to the main engine, just not something I've had the time to refactor yet.

nunit / nunit-console