search

nunit / nunit-console

NUnit Console runner and test engine
MIT License
216 stars 152 forks source link

"Runtimes" flag: running a test suite on multiple runtimes #699

Closed ChrisMaddock closed 2 years ago

ChrisMaddock commented 5 years ago

Currently if you want to run your test suite on multiple runtimes, you have to create a whole bunch of manual 'structure' in your CI process or test suite. For ref, see here: https://github.com/nunit/nunit/issues/3223. This becomes even more prevalent with .NET Standard being specifically designed to run across different runtimes. But what if the console could do all that for you...

I'd like to be able to maintain a single test assembly, and then specify which runtimes I want to run this test assembly on on the NUnit Console command line. The console will then expand my project, and run across all relevant runtimes as if I'd passed in multiple assemblies.

For example, taking the .NET Standard NUnit Engines's own tests, on Windows, we'd ideally want to test on netfx CLR v4, .NET Core 2 (and 3?) and mono. (Currently, we only test on .NET Core.) So I'd like to be able to simply specify something like the below (format tbc!), and then let the console take care of all the hard work:

nunit3-console nunit.engine.tests.dll -runtime netfxclr4, mono, netcore2, netcore3

Further down the line, I'd also like to have extensibility for adding custom test agents for user-registered runtimes. (e.g. a Blazor test agent.) Agent-extensibility is a whole separate ticket, however!

What do people think? Technically, there's nothing stopping us doing this now and building on it as more platforms become available...but I'd like to do more work on the .NET Core console and the refactoring of the RuntimeFramework* classes first. 🙂

CharliePoole commented 5 years ago

Nice... maybe also think about setting the runtime in an NUnit project file. I think it's supported but maybe not for all current runtimes. That way, you could just add the same assembly multiple times with different runtimes.

ChrisMaddock commented 5 years ago

Yes, I hadn't considered that! That would work. 🙂 Hopefully this feature could take away the overhead of maintaining an NUnit project file though - at least for the simple cases.

ChrisMaddock commented 2 years ago

Closing this one. I still think it would be a great feature, but it needs someone to take it on. And right now, that's not me. 🙂