search

nunit / nunit-vs-adapter

Runs NUnit V2 tests inside the Visual Studio 2012 or later Test Explorer window.
MIT License
50 stars 43 forks source link

Avoid UnsupportedFrameworkException case of Nunit3 Tests #118

Closed hometm closed 8 years ago

hometm commented 8 years ago

If Nunit2 tries to run a Nunit3 assembly, a UnsupportedFrameworkException is thrown (insude Nunit 2.6.4). Due to this, VS interprets Nunit3 tests (performed by Nunit2) as 'failed'.

This fix leads to, that VS (running Nunit2) ignores Nunit3 test results.

CharliePoole commented 8 years ago

The purpose of the exception is to ensure that the NUnit VS adapter does ignore NUnit 3 tests. NUnit 3 tests are handled by the NUnit 3 VS adapter, which is a different piece of software entirely.

NUnit 2 cannot run NUnit 3 tests correctly. However, in some cases, it appears to run them, leading to confusion. That's why the exception was added to NUnit 2.6.4 - so that users would not think their tests were being run properly when they really were not.

cklutz commented 8 years ago

Actually, we have the same issue using both NUnit2 and NUnit3 adapters from within TFS (Build). That is, both adapters (and their accompanying DLLs, as well as ingore.addins for the 3.x adapter) are in the controller's custom assemblies directory.

We have a test project setup, that includes one test assembly for each of the following:

All tests in all assemblies are executed correctly, but there is an additional error message:

Exception NUnit.Core.UnsupportedFrameworkException, Exception thrown executing tests in C:...\DemoLibrary.NUnit3.Tests.dll.

I tracked down the issue to exactly the same try/catch block that this CR handles (see also https://github.com/cklutz/nunit-vs-adapter).

Looking at nunit.core.dll, it looks like it throws an UnsupportedFrameworkException when it finds tests for NUnit 3.x (https://github.com/nunit/nunitv2/blob/master/src/NUnitCore/core/Builders/TestAssemblyBuilder.cs#L228), but I would have thought that it then is in turn the job of the adapter to convert that to something non-fatal, if it can.

It already does so during the discovery phase (https://github.com/nunit/nunit-vs-adapter/blob/master/src/NUnitTestAdapter/NUnitTestDiscoverer.cs#L77), but given the issue seen, it looks like it also needs to do so during test execution, shouldn't it?

(Or discovery itself has an issue, that those tests not discovered "for Nunit 2" should be executed.)

CharliePoole commented 8 years ago

I see your point. My assumption was that, no tests having been found, VS would not ask us to execute that assembly. Apparently that's not the case!

The way that discovery and execution are called by VS makes them have to operate in a vacuum. So this update is actually needed. Wish I had seen that before the release!

I'll have to see if I can re-open this. Otherwise, it's an easy fix to insert.

CharliePoole commented 8 years ago

@hometm Seems like your fix was actually needed after all. I thought we had already fixed this, but as explained in the last comment, it's needed in two places. Since your repo has been deleted, I'll just copy and paste the change into my local build and merge it. Thanks for your work.