Open CharliePoole opened 5 days ago
As background for this discussion, here is a list of the ways nunit tests may be run using the packages we create and distribute. Third-party runners are not listed since they will not influence this decision, at least as far as we know.
NUnitLite allows self-executing tests to run and produce output similar to the console runner. The test assembly itself is the executable, so one assembly at a time is run. It executes under the target framework specified by the user and makes use of the standard facilities of .NET to locate any dependencies needed.
The "standard" runner, NUnit.ConsoleRunner, executes under the .NET Framework. It can run multiple test projects in a single execution, including tests targeting multiple frameworks, and produces a single XML result file for all tests run. By default, it launches agent processes to execute tests. We currently bundle agents for .NET 4.6.2, .NET 6.0, .NET 7.0 and .NET 8.0. It can also run .NET Framework tests in-process, which makes debugging simpler..
The "netcore" runner, NUnit.ConsoleRunner.NetCore may be installed as a tool and runs using the command dotnet nunit
. It runs under .NET 6.0 or under a higher runtime if 6.0 is not present. It can execute tests targeting any version of .NET Core equal to or lower than the version under which it runs. It runs only one assembly at a time and the tests always run in the same process as the runner itself.
A new experimental runner, NUnit.ConsoleRunner.Net80, was provided in the 3.18.3 release. It works the same as the "netcore" runner but targets .NET 8.0. It runs using the command dotnet nunit-net80
.
The NUnit3.VisualStudio.Adapter is used to run tools in several ways
vstest.exe
, although this is now somewhat obsolete.dotnet test
command.This issue proposes to eliminate 3 and 4, retaining runners 1, 2 and 5.
@manfred-brands Could you expand on your comment regarding the new MS test runners? My understanding is that this is a runner for tests using the MSTest framework. Not so?
@CharliePoole I can answer that. And No, it is not, it is a general new testing platform. It actually does the same as we do with NUnitLite, so that the test project becomes self-executing. In order to hook this up to the existing infrastructure, they have added and adapter into the NUnit3TestAdapter, which I will try to release a beta of as soon as I can. The test project is still selfexecuting, but will actually work through the NUnit3TestAdapter to hook into Visual Studio and dotnet test.
They have done similar with MSTest and I do believe also XUNit is on the same track now.
Over time NUnit should have its own native implementation of this new testplatform, and we could use NUnitLite as a base for that - althought @manfred-brands said earlier he would like to take a fresh look at NUnitLite and modernize it first.
@OsirisTerje I can't find any link to the general platform, just the MSTest blog post describing executable tests as a new and brilliant idea they just invented. :-) Is the more general thing publicly announced anywhere?
Hi @CharliePoole
thanks for tagging and informing me. Eliminating 3 would also mean eliminating the .net version specific agents and eliminating the option to create an ITestRunner using TestEngineActivator.CreateInstance().GetRunner(new TestPackage(dllfile)) ?
Of course, this would completely brake my use case. In my use case, I create a console application which creates an ITestRunner as mentioned above. Let me give some insights regarding my motivation and use case and why not using your agent or console:
Edit: After re-reading your summary and my comment and discussing with my colleagues, I assume if you still provide the agents targeting net8.0 in Option 2 (and net9.0 ;) ), I still can use my own agent. Please inform me as soon as a working dev build is available. I will test it asap and give feedback.
Thank you @CharliePoole for adding me.
I share @OsirisTerje's opinion on that for the most part. It's straightforward to install, and it would be sad if the .NET tool were gone.
It's very convenient for our case to have it. With 18 Windows machines running UI tests, all we need to do to update the NUnitConsole, is to bump up the version number in a tool manifest of repo. It's also not an issue to go back to a previous version when needed. Without it, we either have to work out an automatic way to provision all the machines (current and new) with a version of NUnitConsole, or update the Windows installation image which is not a straightforward process.
@MichalMucek For my understanding, why can you not use dotnet test
? What does the console runner bring that dotnet test
does not?
@Sputnik24 This issue proposes to remove the .NET Core builds of the console runner. It does not affect the engine at all. It also doesn't affect the existing "standard" runner (i.e. the console runner currently built under .NET framework which uses agents to run tests). Users would be left with one less way to run tests, but all targets currently supported would be supported. Those using the dotnet CLI would lose dotnet nunit but continue to use dotnet test, which relies on the adapter and the agent. We're looking for input from those who would potentially suffer from that change before making a final decision, but your use case of creating your own runner would not be affected.
It's a bit off topic for this issue, but fwiw the existing console runner (option 2) also reverses the typical client server model. It's agents are clients, which connect back to it for instructions. Communication is via TCP or Remoting at this time but the design allows alternate protocols to be added.
@MichalMucek Same question as @manfred-brands about dotnet test. In my case, I ask without a specific opinion. I'm not a user of either but they look the same to me in terms of capabilities. Additionally, the full runner (perhaps that's the term I should use rather than standard) has a greater set of capabilities than our dotnet runner. It's drawback, of course, is that a recent version of the dotnet framework must be installed in order to use it even if all your work is under dotnet core.
From the development point of view, I think it would be better to spend time migrating the standard runner to .NET 8.0 or higher rather than continuing to maintain 2 or 3 different builds of the runner each with different capabilities. Of course, if more people were working on this, that may be less of an issue.
@Sputnik24 It's a bit off topic for this issue, but fwiw the existing console runner (option 2) also reverses the typical client server model. It's agents are clients, which connect back to it for instructions. Communication is via TCP or Remoting at this time but the design allows alternate protocols to be added.
I know, that's why I mentioned that I reversed the client/server logic of your agent and had to develop my own agent being the server. So, to sum up, I would not be affected by this change.
Again, thanks a lot for informing me.
@manfred-brands @CharliePoole
@MichalMucek For my understanding, why can you not use
dotnet test
? What does the console runner bring thatdotnet test
does not?
Better filtering. It allows filtering the SpecFlow BDD tests by the examples. It's convenient and useful because different examples may have different categories. dotnet test
does not have such a capability. It can at most run scenarios with a whole batch of examples.
It allows filtering the SpecFlow BDD tests by the examples.
I am not a Specflow guy. Can you show how you call the console "by examples"?
@MichalMucek
The filtering issue makes me want to think a bit more. It would be good to have examples, including those that work with both dotnet test
and dotnet nunit
and those that only work with dotnet nunit
.
Is filtering any different using SpecFlow vs NUnit by itself?
@OsirisTerje To be precise, I mean filtering and listing (exploring in terms of NUnit).
Let's consider the below test case in the Gherkin language based on an imaginary PrinerHelper application:
Scenario Outline: PrintingHelper can connect to a printer
Given PrinterHelper was started
And "Color" printer was connected to PC using "<CableType>" cable
When User clicks printers tab
Then user will see "Color" printer on the list
@USB
Examples:
| CableType |
| USB |
@Serial
Examples:
| CableType |
| Serial |
Having two PCs, each with a printer connected using a different cable (PC1 - USB; PC2 - Serial). We want to run a specific example on a particular PC. Otherwise, the scenario will fail when running every example on each machine. To resolve it, we have to run each example separately and NUnitConsole allows that. Let's say there are about 800 scenarios with similar examples and we add more PCs to speed up the execution.
With the below command, we're able to obtain the IDs (they're the key) and test method names of scenario examples with @Serial category. Obtained list can be split into smaller lists that could run in parallel on multiple PCs/agents.
dotnet tool run nunit tests.dll --test-name-format="{i} {M}" --noheader --where cat==Serial --explore
dotnet test --list-tests
can list all the test scenarios with their examples, but that's about it. --list-tests
doesn't work with --filter
. However, the --filter
works alone, but then again, it's not possible to split it into smaller lists that run in parallel.
Edit: I forgot to mention that SpecFlow is responsible for generating C# code with test methods from the scenarios written in Gherkin language. Test steps are bonded with methods written in C#.
@MichalMucek OK, that's a good example although the translation to C# is a bit of a black box I wish I understood better.
What I'm missing is this... what prevents you from using the standard console runner to generate your test lists? Also, do you execute the tests using the SpecFlow runner or the console?
@CharliePoole
What I'm missing is this... what prevents you from using the standard console runner to generate your test lists? Also, do you execute the tests using the SpecFlow runner or the console?
To be fair, there is no actual blocker for us. We prefer using the .NET tool because of its ease of installation in our test environment. Implementation-, maintenance-, and time-wise it is cheaper. Relying on the standard executable would mean either a few hours spent each time when updating the Windows installation image with a new version of the Console, or creating a custom way of provisioning automatically Windows machines with NUnitConsole. Chocolatey is a no-go for us in our test environment due to network infrastructure restrictions.
We execute the tests using the Console. SpecFlow alone can't do that. https://docs.specflow.org/projects/specflow/en/latest/Execution/Executing-SpecFlow-Scenarios.html
BTW SpecFlow development is practically abandoned. It has been taken over by fork project Reqnroll.
@MichalMucek
That clears up some things for me, but the looking at the SpecFlow docs confused me. :-(
Their docs state that for test discovery and execution, you need
That implies use of dotnet test
or the Visual Studio IDE rather than dotnet nunit
. So are the docs out of date?
Thanks for the info about Reqnroll. I'll take a look at that - maybe the that team will be a bit more transparent than SpecFlow was since they are eliminating the non-open components.
I'm reaching the conclusion that we can't really get rid of the tool at the moment for two main reasons:
dotnet test
I'd like to do this in a way that minimizes the level of effort we put in, so I'm suggesting the following for 3.19...
The following items are more properly part of issue #1492, so I'll repeat and enlarge on them there.
It may also be helpful to consider adding better filtering to our support of dotnet test
.
What does everyone think of the above as a resolution?
I believe what you write is sound. I agree.
But
I still don't understand why it is said that dotnet test
can't filter the same way as the console. It might be true that the built in filter in dotnet test
doesn't have that capability, but dotnet test
calls on the adapter which again calls the engine, and you can actually pass any NUnit filter down there from dotnet test
. The statement to do this is the NUnit Where filter statement. See https://docs.nunit.org/articles/vs-test-adapter/Tips-And-Tricks.html#where and this blogpost too: https://blog.prokrams.com/2019/12/16/nunit3-filter-dotnet/ Also note that the Where filter can be passed down as a command line argument.
I'll move on to issue #1492 and continue to build out a plan for 3.19. It looks like this one can be closed, but I'll let it sit for a few days to see if there are any other responses, esp. from @MichalMucek regarding the --where filter.
Discussion under #1493 have led to this decision. I'll excerpt some of it here so all the discussion can be in one place.
@manfred-brands commented:
@CharliePoole commented:
@OsirisTerje commented:
@CharliePoole commented:
This is an important decision so we want everyone affected to have a chance to express their opinions. In addition to @OsirisTerje and the rest of the @NUnit team, the following folks are mentioned because they were recently involved in discussions about the netcore runner: @MichalMucek @kogoel @bouchraRekhadda @Sputnik24
Of course, anybody who uses it is welcome to comment.