search

nunit / nunit3-vs-adapter

NUnit 3.0 Visual Studio test adapter for use under VS 2012 or later
https://nunit.org
MIT License
203 stars 105 forks source link

Version 3.0.8.0 as nuget package only. No tests detected in Visual Studio 2015 #117

Closed RobertK66 closed 8 years ago

RobertK66 commented 8 years ago

I am using Microsoft Visual Studio Professional 2015/Version 14.0.24720.00 Update 1.

Steps to reproduce: 1) Disable or Uninstall the Extension "Nunit3 Test Adapter" in Visual Studio 2) Create new Solution with C# ClassLibrary targeting .net Framework 4.6.1 3) Add Nuget package Nunit .3.0.1 4) Write simple test and Compile. 5) Add "Nunit3TestAdapter" latest prerelease 3.0.8-ctp-8 6) Rebuild Solution

-> Test is not found. Log shows: ------ Discover test started ------ NUnit Adapter 3.0.8.0 discovering tests is started ========== Discover test finished: 0 found (0:00:00,3490436) ==========

7) Change Nuget package to NunitTestAdapter 3.0.7-ctp-7b 8) Rebuild shows: ------ Discover test started ------ NUnit Adapter 3.0.8.0 discovering tests is started NUnit Adapter 3.0.7.0 discovering tests is started NUnit Adapter 3.0.7.0 discovering test is finished ========== Discover test finished: 0 found (0:00:01,2087276) ==========

9) Close VS / Reopen now catches and shows the test in TestExplorer ========== Load Playlist finished (0:00:00,0156266) ========== ------ Discover test started ------ NUnit Adapter 3.0.7.0 discovering tests is started NUnit Adapter 3.0.7.0 discovering test is finished ========== Discover test finished: 1 found (0:00:04,4643586) ==========

When a project uses the 3.0.8.0 Version of the nuget package in its dependencies this prevents any tests to be found by the (my?) Visual Studio Test Explorer. (It does work on my TFS build!) Even installing the "Nunit3 Test Adapter" Extension does not resolve this issue. Only when I go back to the nuget package 3.0.7.0 then it works independend of an VS Extension to be available or not. The log if both are there looks like this:

------ Load Playlist started ------ ========== Load Playlist finished (0:00:00,0049996) ========== ------ Discover test started ------ NUnit Adapter 3.0.7.0 discovering tests is started NUnit Adapter 3.0.8.0 discovering tests is started NUnit Adapter 3.0.7.0 discovering test is finished NUnit Adapter 3.0.8.0 discovering test is finished A test with the same name 'ClassLibrary1.Class1.Test' already exists. This test is not added to the test window. ========== Discover test finished: 2 found (0:00:05,7947319) ==========

snboisen commented 8 years ago

I am facing the same problem with CTP-8 NuGet package and VS2015 (Community Edition). No tests discovered. I have never had NUnit Test Adapter in any version installed in this Visual Studio as an extension.

thantos commented 8 years ago

Same issue here, VS 2015 Pro. Win 7. Simple solution, no tests discovered. V2.0.0.0 discovers the test, but gives a framework version error (as expected).

Tankatronic commented 8 years ago

Also having this issue. Win7 64 bit using VS 2015 Professional Update 1 using NUnit v 3.0.1 and NUnitTestAdapter v 2.0.0

Installing the NUnitTestAdapter.WithFramework v2.0.0 will discover tests, but all tests are run twice.

Using NUnit v 3.0.1 with NUnitTestAdapter v 1.2.0 will discover tests, but all tests are run twice.

OsirisTerje commented 8 years ago

@Tankatronic : Do you also have this with the NUnit3 adapter ?

Tankatronic commented 8 years ago

@OsirisTerje With the Nunit3TestAdapter v3.0.8-ctp-8 and NUnit v3.0.1 no tests are discovered.

Tankatronic commented 8 years ago

@OsirisTerje Had some odd behavior just now, not sure if this will aid in debugging but thought I would document it. I was no longer able to discover tests after installing Nunit3TestAdapter even after reverting to a previously working configuration with NUnitTestAdapter v1.2.0. I was able to resolve the issue by restarting Visual Studio - I did not have to do this in any other situation. Here are the steps to reproduce:

  1. Install NUnit v3.0.1
  2. Install NUnitTestAdapter v1.2.0
  3. Build and verify tests are discovered.
  4. Uninstall NUnitTestAdapter v1.2.0
  5. Install NUnit3TestAdapter v3.0.8-ctp-8
  6. Build - No tests discovered
  7. Uninstall NUnit3TestAdapter v3.0.8-ctp-8
  8. Install NUnitTestAdapter v1.2.0
  9. Build - No tests discovered
  10. Restart Visual Studio
  11. Build - Tests discovered

I can uninstall the NUnitTestAdapter and reinstall v1.2.0 or update it to version 2.0.0 and then revert back to v1.2.0 and not have to restart Visual Studio inbetween to rediscover tests. This is all from the Nuget Package Manager.

rprouse commented 8 years ago

@Tankatronic one thing is odd in your steps, you said that in step 3 that tests were discovered. The old adapter should not discover tests compiled against NUnit 3. Did you update the references in your project to NUnit 3?

Similarly, the v3 adapter will not discover tests if your projects reference NUnit 2.

Tankatronic commented 8 years ago

@rprouse I am getting really strange issues now.

I have been using NUnit v3.0.1 for these tests. I have verified that it is the only package installed on my project. I went to verify that the extension I had previously installed was removed entirely and I found the following files: NUnit.2.6.4.nupkg and NUnitTestAdapter.2.0.nupkg inside C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\Extensions\Xamarin\Xamarin\3.11.816.0\Packages. It seems these files were not removed when I uninstalled the extension using the Extension manager and VS was still using them.

Now is where it gets weird.

  1. I removed those two files and restarted VS. I had NUnit v3.0.1 and NUnitTestAdapter v 1.2.0 installed. My tests were discoverable.
  2. I updated to NunitTestAdapter v 2.0.0 and my tests were still discoverable (this was a first).
  3. I restarted Visual Studio without making further changes and my tests were no longer discoverable after restart until I downgraded back to NUnit v.1.2.0.

I was able to repeat this several times. I then tried the following.

  1. I installed NunitTestAdapter v 1.2.0. My tests were discoverable.
  2. I updated to NunitTestAdapter v 2.0.0. My tests were discoverable.
  3. I uninstalled the NUnitTestAdapter. My tests were discoverable.
  4. I restarted Visual Studio. My tests were not discoverable.

Here is the Test output during discovery After step 1:

------ Discover test started ------ NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started Attempt to load assembly with unsupported test framework in filePath NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit 1.2.0.0 discovering tests is started NUnit 1.2.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit 1.2.0.0 discovering test is finished NUnit 1.2.0.0 discovering test is finished ========== Discover test finished: 16 found (0:00:02.7750811) ==========

Here is the Test output during discovery After step 2:

------ Discover test started ------ NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started Attempt to load assembly with unsupported test framework in filepath NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit 1.2.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit 1.2.0.0 discovering tests is started NUnit 1.2.0.0 discovering test is finished NUnit 1.2.0.0 discovering test is finished ========== Discover test finished: 16 found (0:00:06.450645) ==========

Here is the Test output during discovery After step 3:

------ Discover test started ------ NUnit 1.2.0.0 discovering tests is started NUnit 1.2.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started Attempt to load assembly with unsupported test framework in filePath NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit 1.2.0.0 discovering test is finished NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit 1.2.0.0 discovering test is finished ========== Discover test finished: 16 found (0:00:00.6914798) ==========

Here is the Test output during discovery After step 4:

------ Discover test started ------ NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started Attempt to load assembly with unsupported test framework in C:\justinProjects\development\Nuget Packages\mccoyscore\McCoys.Tests\bin\Debug\McCoys.Tests.dll NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit VS Adapter 2.0.0.0 discovering test is finished ========== Discover test finished: 0 found (0:00:01.196402) ==========

I am not sure where the version 2 adapter is still being referenced from. In step 4 my project only referenced nunit.framework, but it still appeared the the v2 test adapter was still there somewhere.

rprouse commented 8 years ago

Odd, in all those cases, only the NUnit 2 adapter is installed. I don't see in those steps where you install the NUnit 3 adapter which is used to discover and run NUnit 3 tests. It is version 3.0.8 and is downloaded from https://visualstudiogallery.msdn.microsoft.com/0da0f6bd-9bb6-4ae3-87a8-537788622f2d

rprouse commented 8 years ago

Very odd since you were using the 3.0.8 adapter before. Have you tried 3.0.8 as an installed extension rather than a NuGet package? It should live side by side with the 2.0 adapter.

CharliePoole commented 8 years ago

@rprouse We have reports of the side-by-side installation no longer working in VS2015. I have not yet verified.

On Mon, Feb 1, 2016 at 2:12 PM, Rob Prouse notifications@github.com wrote:

Very odd since you were using the 3.0.8 adapter before. Have you tried 3.0.8 as an installed extension rather than a NuGet package? It should live side by side with the 2.0 adapter.

— Reply to this email directly or view it on GitHub https://github.com/nunit/nunit3-vs-adapter/issues/117#issuecomment-178223933 .

Tankatronic commented 8 years ago

After installing the Nunit3 Test Adapter v 3.0.8 via the Extension Manager I am able to discover tests. Here is the Test Discovery output

------ Discover test started ------ NUnit Adapter 3.0.8.0 discovering tests is started NUnit Adapter 3.0.8.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started Assembly contains no NUnit 3.0 tests: filePath to non unit test project NUnit Adapter 3.0.8.0 discovering test is finished Attempt to load assembly with unsupported test framework in filePath1 to unit test project NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit Adapter 3.0.8.0 discovering test is finished NUnit VS Adapter 2.0.0.0 discovering test is finished ========== Discover test finished: 16 found (0:00:01.5837915) ==========

Still have no clue why the version 2 adapter is hanging around. Any suggestions on where I may find it?

I also tested this using the Nuget Package. I was not able to discover tests. Here is the output:

------ Discover test started ------ NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit Adapter 3.0.8.0 discovering tests is started Attempt to load assembly with unsupported test framework in filePath NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit Adapter 3.0.8.0 discovering tests is started ========== Discover test finished: 0 found (0:00:02.7030808) ==========

CharliePoole commented 8 years ago

You appear to have the 2.0 adapter installed as well as 3.0. These are two separate programs, one called NUnit VS Adapter and the other NUnit3 VS Adapter. If the 2.0 adapter is installed via nuget, you can examine the nuget packages in your solution and uninstall it. If it is installed globally, via a vsix extension, remove it using Tools | Extensions and Updates.

Of course, if the 2.0 adapter is installed globally for use by other projects that use NUnit 2.6.4, then those tests will no longer be detected once you remove the extension.

RobertK66 commented 8 years ago

In my original post, you can see that in my case there is no Version 2.0.0.0 interfering with the detection at all. I did use Nunit in other projects before but I do not have the old version of the VSXI Adapter in my Extensions List. I also crosschecked with another VS installation on the same machine (VS 2013 – Community) but there is also no NUnit Adapter VSXI Plugin installed there. As this is a long living developer machine, I am not sure if there was a time where I had one of the older predecessors of the Plugin installed somewhere but I cannot find any references to this now.

@Tankatronic: Testing this issue you always have to restart Visual Studio. It seems that the first version of the adapter used and loaded cannot be overridden/unloaded by simply changing the nugget package in the solution (see my Step 8 – 9). This IMHO also causes that in my case the VSXI Plugin together with the nuget package in the solution does not work. If the (broken?) nuget package is loaded first, it prevents the plugin from working correctly. Only VSXI 3.0.8.0 together with nuget 3.0.7.0 seems to work. (However, with this configuration I found that tests are executed twice shown under one entry in TestExplorer….)

@CharliePoole: I am not intending to have a side by side instalation of the VSXI Plugin versions. It seems that I am not able to use the newest version on its own. Do you have some advice where I should look for possible debris of unsuccessful prior Install/Uninstall events?

RobertK66 commented 8 years ago

I found the problem! I managed to download your latest source and compiled a version with the LAUNCHDEBUGGER option. Then after some debugging, I could nail down the Problem to the following Exception:

NUnit Adapter 3.0.8.3 Exception: System.ArgumentException: Illegales Zeichen im Pfad. bei System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional) bei System.IO.Path.Combine(String path1, String path2) bei NUnit.Engine.Services.ExtensionService.ProcessAddinsFile(DirectoryInfo baseDir, String fileName) bei NUnit.Engine.Services.ExtensionService.FindExtensionsInDirectory(DirectoryInfo startDir) bei NUnit.Engine.Services.ExtensionService.StartService() bei NUnit.Engine.Services.ServiceManager.StartServices() bei NUnit.Engine.TestEngine.Initialize() bei NUnit.Engine.TestEngine.GetRunner(TestPackage package) bei NUnit.VisualStudio.TestAdapter.NUnitTestAdapter.GetRunnerFor(String assemblyName) in C:\Develop\ws_ext\NunitAdapter\nunit3-vs-adapter-master\src\NUnitTestAdapter\NUnitTestAdapter.cs:Zeile 118. bei NUnit.VisualStudio.TestAdapter.NUnit3TestDiscoverer.DiscoverTests(IEnumerable`1 sources, IDiscoveryContext discoveryContext, IMessageLogger messageLogger, ITestCaseDiscoverySink discoverySink) in C:\Develop\ws_ext\NunitAdapter\nunit3-vs-adapter-master\src\NUnitTestAdapter\NUnit3TestDiscoverer.cs:Zeile 48.

-> BINGO. That gave enough hints to find the Bug and a solution/workaround:

If you install the Version 3.0.8.0 as a Nuget package then there is a corrupted (not empty!) “ignore.addins” file installed in the packages\NUnit3TestAdapter.3.0.8-ctp-8\lib folder. Earlier Versions did install a 0byte empty file!

After removing the content of this file in the <solution>/packages/NUnit3TestAdapter.3.0.8-ctp-8\lib folder and deleting the cached version of the adapter under <username>\AppData\Local\Temp\VisualStudioTestExplorerExtensions... everything works as expected!

Tankatronic commented 8 years ago

@CharliePoole You were correct, I had the NUnitTestAdapter v 2.0 installed at the solution level. Not sure if it was installed using a previous version of nuget that had a concept of solution level packages or not. It was not showing up as installed under the Tools | Extensions menu or in any of the projects Nuget packages. Either way, deleting the package and restarting visual studio resolved that particular issue.

Here are my new discover results for version 1, 2, and 3 of the test adapter installed using NuGet. All of these tests were using NUnit3. The Nunit3 Test Adapter installed via the extension manager still works, but the v 1.2.0 test adapter is the only nuget package that will discover tests with NUnit3.

V 3 (Extension):

------ Discover test started ------ NUnit Adapter 3.0.8.0 discovering tests is started Assembly contains no NUnit 3.0 tests: filePath NUnit Adapter 3.0.8.0 discovering test is finished NUnit Adapter 3.0.8.0 discovering tests is started NUnit Adapter 3.0.8.0 discovering tests is started NUnit Adapter 3.0.8.0 discovering test is finished NUnit Adapter 3.0.8.0 discovering test is finished ========== Discover test finished: 19 found (0:00:17.2917921) ==========

V 3 (Nuget package):

------ Discover test started ------ NUnit Adapter 3.0.8.0 discovering tests is started NUnit Adapter 3.0.8.0 discovering tests is started ========== Discover test finished: 0 found (0:00:00.6190619) ==========

V 2:

------ Discover test started ------ NUnit VS Adapter 2.0.0.0 discovering tests is started NUnit VS Adapter 2.0.0.0 discovering tests is started Attempt to load assembly with unsupported test framework in filePath NUnit VS Adapter 2.0.0.0 discovering test is finished NUnit VS Adapter 2.0.0.0 discovering test is finished ========== Discover test finished: 0 found (0:00:02.0482048) ==========

V 1:

------ Discover test started ------ NUnit 1.2.0.0 discovering tests is started NUnit 1.2.0.0 discovering tests is started NUnit 1.2.0.0 discovering test is finished NUnit 1.2.0.0 discovering tests is started NUnit 1.2.0.0 discovering test is finished NUnit 1.2.0.0 discovering test is finished ========== Discover test finished: 19 found (0:00:01.2691269) ==========

Tankatronic commented 8 years ago

@RobertK66 Removed my original post. I needed to delete the entire cached version of the extension, not just the ignore.addins file.

The nuget extension now works properly after following your instructions.

RobertK66 commented 8 years ago

ok, changed my post above to make workaround clearer, thx.

RobertK66 commented 8 years ago

This is weird. Look at the difference between nuget packages version 3.0.8.0 and 3.0.7.0 (ctp-7b) opened in 7Zip:

https://cloud.githubusercontent.com/assets/16986510/12757898/3eb2c220-c9da-11e5-90ca-656dfd615f81.png

ctp8 changed the 'Verfahren' (I assume this should be 'compression method') from 'deflate' to 'store'. Using 7Zip to uncompress does the right thing (-> 0byte file) - using Explorer Unzip under windows 10 does not generate this file at all (!?) and - as mentioned above - using the NUGET Package Installer from Visual Studio 2015 generates a file containing 2 unprintable bytes!

Maybe you should consider to put something in there (…a commented line showing what could be achieved with a 'ignored addin' !?...) to avoid such a mess generated by inconsistent tool chain ;-)...

CharliePoole commented 8 years ago

I created a new issue #118 for fixing the addins file.

To be honest, I'm getting a little confused about what remains to be resolved in this issue, if anything, but that one I can do something about. :-)

RobertK66 commented 8 years ago

I am perfectly happy with #118, thx. The issues brought up by @Tankatronic about mixing and intermingling the different versions and frameworks are beyond my scope ;-).

jsandquist commented 8 years ago

I stumbled upon this same issue and commented on another user's experience on stackoverflow a week ago.

118 will most likely fix this and #117 be closed with it.

Surprisingly, today no tests were detected (again) but it turned out this file was accidentally gone (I probably had deleted it but forgot about it...) <username>\AppData\Local\Temp\VisualStudioTestExplorerExtensions\NUnit3TestAdapter.3.0.8-ctp-8\lib\ignore.addins

Adding an empty ignore.addins or removing the VisualStudioTestExplorerExtensions folder fixed it as indicated earlier by @RobertK66 .

ferpega commented 8 years ago

@RobertK66 indications have resolved "tests not detected" issue here too. With Visual Studio 2015 Version 14.0.24720.00 Update 1

RobertK66 commented 8 years ago

As this original issue is still open, I use it now to add a reference to my suggestion on improvement for fix https://github.com/nunit/nunit3-vs-adapter/issues/118 here: https://github.com/nunit/nunit3-vs-adapter/issues/118#issuecomment-201750419

CharliePoole commented 8 years ago

This appears to have been fixed in conjunction with the last PR for issue #118. Once the release is out, please give us some feedback in case there's still a problem.

RobertK66 commented 8 years ago

issue is fixed now (CTP9) - also retested ok with Visual Studio 2015 Update 2.

CharliePoole commented 8 years ago

Great! Thanks for letting us know.

meghanegi15 commented 6 years ago

Do you know which version of nunitlite can I use for WindowsEmbeddedCompact,Version=v3.9? I am looking forward to write unit test for this framework.

rprouse commented 6 years ago

@meghanegi15 your comment would be better served as a new issue. It doesn't really belong here, but to answer your question, I believe that you need to use NUnit 3.5 or older. We dropped support for Compact Framework in 3.6 and to be honest, I'm not even sure our CF builds will work with Windows Embedded Compact, but download the CF ZIP files from https://github.com/nunit/nunit/releases/tag/3.5 and let us know.