Create .NET Core Console Runner

[ChrisMaddock]

A simple .NET Core console runner, utilising the existing .NET Standard engine. This will initially be a restricted version of the main console, due to limitations of the .NET Standard engine. Over time, we hope both engine and console will develop, and we will have near feature parity (where appropriate!)

Blocked by: https://github.com/nunit/nunit-console/issues/388

[CharliePoole]

If there's a specific use case or cases for why we want to do this, I think it helps to add it to the issue for those of us who don't know what the cases are. :smile:

[ChrisMaddock]

The primary use case would be the ability to run NUnit tests on platforms where neither the .NET Framework or Mono are installed. 🙂

The only way we can currently do that is via dotnet test. As we know, running via the VS adapter has a number of restrictions - having our own runner allows us to not be subject to these.

[CharliePoole]

So, for example, on Linux? Allowing the user to avoid Mono? Is there a call for that?

I write this question in part for my own purposes. I've tried to always give preference to using Mono when there is an option. OTOH .NET Core 3.0 is starting to look pretty usable at long last.

[ChrisMaddock]

Absolutely - at least where I work! 😄

I disagree with "3.0 is starting to look pretty usable" - we've been running 2.x in production for a year now. Mono doesn't come into our picture - we're running on Linux machines which only have .NET Core installed. I don't want to have to install mono in to that environment, just to run the NUnit Console, and then in turn have that spin up a .NET Core process, when it could just run in .NET Core in the first place. 🙂

[CharliePoole]

Bear in mind that I'm totally a desktop developer in .NET. Integration of SWF and WPF make it an option now.

[rprouse]

We are investing in .NET Core heavily too. All new server side development is being done in .NET Core for portability and performance. Mono doesn't come into the picture anymore except for mobile development with Xamarin. Mono isn't installed on any of our Linux servers. 1.4 was usable for most ASP.NET apps, but 2.0 hit the sweet spot for us. I am looking forward to 3.0 to start migrating some of our legacy code.

[CharliePoole]

Yes it's clear that's the way things will be going - and already have to some extent. I'm a bit mystified about why needing to install Mono is seen as a bad thing - I do it on every Linux system I have.

My new work is all in .NET Core, but that's not ready for release and won't be for a while. However, I remain dubious about the value of a .NET Standard engine as opposed to .NET Core.

If 3.0 proves usable, TestCentric 2.0 may use it. Otherwise, I'm looking at a third-party front end that generates code for multiple GUI platforms.

[jnm2]

@CharliePoole Windows Forms and WPF are optional Windows-only libraries that sit on top of .NET Core 3.0, which itself is cross-platform. For UI on other platforms, you'd need Windows Forms on Mono or some other GUI-focused community project.

[jnm2]

Of all the .NET runtimes, .NET Core is the only one in the last several years that is getting significant runtime and BCL investments. All the performance improvements, bug fixes, and new features happen there. It's like pulling teeth to get bugfixes ported to .NET Framework (ask me about this sometime 😬).

[CharliePoole]

@jnm2 Sounds like I may have gotten my hopes up too soon.

[Rain9333]

Also its nice if you want to run some tests on Mac :)

[jaydeept]

Hello,

Do you guys have any tentative plan to support .Net Core for NUnit console runner? I am able to execute the test cases using dotnet vstest currently but I really didn't like the generated result file, it's bit complex to parse. I really like the result file generated by nunit-console for .Net Framework.

[jnm2]

@jaydeept Yes, this thread is tracking that plan.

[olegbaslak]

Seems to be not blocked by #388 any more :)

[dayalan]

Hello, just checking if there is now a more formal release date for this?

[ChrisMaddock]

Not yet. NUnit is run entirely by volunteer members of the community - all contributions welcome!

I do actually have a half finished branch that makes a start on this one however, which I’m hoping to get back to soon...

[dayalan]

Appreciate the update. Thanks.

[santhosh-James]

Is there a way to pass the argument like --where cat=SmokeTests --result=TestResult.xml in .dotnetcore from the command line?

[beloquintana]

@santhosh-James, you can use: dotnet test --filter testcategory=SmokeTests. There are others options in https://docs.microsoft.com/en-ca/dotnet/core/tools/dotnet-test?tabs=netcore21.

[ghost]

Hi. I've recently discovered the .netstandard issue when trying to generate a coverage report with OpenCover and nunit3-console as target, which is intended to be pushed to CodeCov at the end of test workflow. Is there any workaround so as to generate coverage reports targeting .netstandard?

[jnm2]

@joaosantana I've used dotnet test and the excellent https://github.com/SteveGilham/altcover which integrates with it for my NUnit tests.

[ghost]

@jnm2 I'll take a look, thanks for suggestion.

[dayalan]

With regards to filtering test fixture categories using Dotnet Test filter command, is it possible to filter multiple categories using a single command? i.e. dotnet test --logger "trx;LogFileName=Test_Execution_Results\results.trx" --filter TestCategory=A&TestCategory=B

[schrufygroovy]

If there's a specific use case or cases for why we want to do this, I think it helps to add it to the issue for those of us who don't know what the cases are. 😄

In our case we are running a big suite of integration tests via nunit console runner, that are already build and prepared. So we can run the same DLL's on various environments.

We don't wanna fetch the solution and build it on every environment, just to execute the tests.

[papatelst]

Temporarily we are using dotnet.exe test command to run tests after migration to .net core. However, I would like help in finding the equivalent to following Nunit command - --params=testkey=testvalue which is then read from code setup at runtime as string value= TestContext.Parameters["testkey"]; How can I do this using dotnet.exe parameters ? Did not find anything here - https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test?tabs=netcore21 Please help.

[ChrisMaddock]

Hi @papatelst - you need to define these in a .runsettings file. The Nunit adapter docs should help you with that.

[papatelst]

.runsettings does not work for us, as it is only for vstest.console.exe command as mentioned here - https://stackoverflow.com/questions/47098041/using-runsettings-file-when-running-nunit-tests-via-command-line We use dotnet.exe test \UITests.csproj command and don't plan to bring in references like TestContext.Properties property which are required to access a property from test as mentioned at link

[ChrisMaddock]

dotnet test also uses the same runsettings file - see the first docs page you linked for details of how to pass it in. 😊

[papatelst]

I tried to use following command - dotnet test UITests\UITests.csproj -s test.runsettings However, I dropped the idea as it requires to bring in references like TestContext.Properties property which are required to access a property from test as mentioned at link

[ChrisMaddock]

The .NET Core Console is now released in Beta! We'd love for people to give it a try and track down any potential problems before release.

The .NET Core console is available on GitHub, and in a new NuGet package named NUnit.ConsoleRunner.NetCore. Please let us know about any problems, and contributions of fixes as well are always welcome!

[ejball]

@ChrisMaddock, would it be possible for NUnit.ConsoleRunner.NetCore to be built as a .NET Core Tool? That was certainly my expectation for a .NET Core console app.

[ChrisMaddock]

@ejball - good question. Two main reasons behind the current position:

1. As you may know, NUnit is maintained entirely by voluntary contributions from the community - there's no paid workers here! Whilst there's been a lot of interest in the outcome of this issue, there's been very few people interested in working on this. From my personal point of view as the person who added the current build - I was very aware how late we are in getting the first version of the console to run .NET Core tests out there - and my key aim was to get an MVP out there first, and build on that. Adding a .NET Core Tool version of the software is not completely off the menu, but a feature which could come later in my opinion.

2. I believe XUnit originally tried to support their test runner under dotnet xunit, and eventually were forced to discontinue that product and instead do an independent executable, as the NUnit Console currently is. I'm haven't followed their issues too closely, but I believe they encountered limitations in the .NET Global Tooling functionality which made it impossible to run tests accurately in all cases - which I think we'd all agree has to be the priority for a test runner!

If someone was interested in owning this idea and taking it forward - I'd be open to it, but keen to make sure we learn from the XUnit team's experience, and only release something that we're sure is accurate and maintainable going forwards. If you fancy digging further - by all means open up a new issue and get stuck in - I'd be interested in the conversation if nothing else!

[ejball]

@ChrisMaddock, FWIW, I believe this is all that is necessary to build it into a local tool: https://github.com/nunit/nunit-console/pull/810

I can't speak to the other potential issues, and I unfortunately can't own the idea and take it forward, but I wanted to see if I could at least get it working as a tool. Thanks for your reply and for your work on NUnit!

[mcichonqa]

Hi, does nunit.consolerunner.netcore\3.12.0-beta1 supported NUnitProject files? I tried execute my tests with NUnitProject.nunit file and I got this:

1) Invalid : some path...\ClassLibrary1\NUnitProjects.nunit
File type is not supported.

[mikkelbu]

I think the problem is that not all of the extensions support .NET Core (yet). To load *.nunit files you need a .NET Core version of the nunit-project-loader. The master build supports .NET Core, but this has not been released to nuget, see e.g. nunit/nunit-project-loader#15