NUnit Test Adapter 2.1 doesn't work with projects that target .NET Framework 3.5 / CLR 2.0

[MaxG117]

Tests for projects that target .NET Framework 3.5 / CLR 2.0 don't run with NUnit Test Adapter 2.1. I saw entries like "'System.BadImageFormatException' in nunit.util.dll" and "'System.InvalidOperationException' in Microsoft.VisualStudio.TestPlatform.TestExecutor.Core.dll" in the Debug window. I'm guessing this is because Mono.Cecil added with #116 targets CLR 4.0.

I had to manually reinstall Test Adapter 2.0 to get my tests to work again. I'm using VS 2015 with SP3, but it's probably irrelevant.

[pvandervelde]

It's not just .NET framework 3.5. I've seen errors appearing when running against .NET 4.5.2 as well. In my case I get:

NUnit VS Adapter 2.1.0.7 executing tests is started
Loading tests from C:\vcs\git\devinfrastructure\Components-Serialization\src\Components.Serialization.UnitTests\bin\Debug\Vista.DevInfrastructure.Components.Serialization.UnitTests.dll
Dependent Assembly  of C:\vcs\git\devinfrastructure\Components-Serialization\src\Components.Serialization.UnitTests\bin\Debug\Vista.DevInfrastructure.Components.Serialization.UnitTests.dll not found. Can be ignored if not a NUnit project.
NUnit VS Adapter 2.1.0.7 executing tests is finished
No test is available in C:\vcs\git\devinfrastructure\Components-Serialization\src\Components.Serialization.UnitTests\bin\Debug\Vista.DevInfrastructure.Components.Serialization.UnitTests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

And then nothing happens. Also having to roll back to 2.0 to make it work.

[OsirisTerje]

@MaxG117 : I can confirm this. What I get is a BadImageFormatException, a AssemblyNotSupportedWarning in the executor. And you're right, the Mono.Cecil dll's are 4.5, 3 out of 4. @pvandervelde : The message you get is different. Can be another variant of the same.

Are you using the vsix or the nuget adapter ?

@CharliePoole @rprouse : The references are pointing to the 3.5 versions, but in the output we get the 4.5 versions. I've checked the packages, and the libs there are correct, so something in VS is grabbing the wrong ones.

Ok, the 4.5 versions are referred in the NunitTestAdapterTest project, and since that are building after the adapter, we get the wrong ones in the common output folder. Also, the installer are in fact referring to the 4.5 ones directly too, since the installer itself is targeting 4.5.

[pvandervelde]

I can open another issue if that makes more sense

[OsirisTerje]

No, you can wait a bit. I will get out a fix within a couple of days. I can send over a nuget package you can test out first.

[OsirisTerje]

NUnitTestAdapter.2.1.1-b2.zip

The attached zip file is a pre-test of the fix. It works on my repro solution here. It would be great if you could check it out too. Unzip the file, and copy the package to a local nuget folder, add that folder to your nuget package sources, and update to this adapter. After the update you need to restart Visual Studio. Please ensure you dont have any Vsix versions of the adapter installed.

In the Output/tests you should now see the version number line below: NUnit VS Adapter 2.1.0.8 executing tests is finished

[CharliePoole]

@OsirisTerje It's a little confusing to me how it could even have been built that way, with a .NET 2.0 assembly referencing a .NET 4.0 assembly.

[pvandervelde]

Ok I'm confused. How do I update to the adapter without the installer? I've unzipped the file and dropped the nuget package in my local nuget feed directory. So how do I update to the adapter from there?

[CharliePoole]

@OsirisTerje Never mind that last - I see your previous comment now.

I had to deal with the same issue in the NUnit 3 adapter. I believe you may need to remove some references. See the NUnit 3 adapter release instructions, which explain how to remove some references each time certain packages are updated. It's annoying but necessary.

[OsirisTerje]

@pvandervelde From your comment it seems you're using the vsix. If you can change your approach for the test only and use the nuget adapter approach instead. Read https://github.com/nunit/docs/wiki/Adapter-Installation and/or Read the section "Providing 3rd party adapters through NuGet" in https://blogs.msdn.microsoft.com/visualstudioalm/2013/06/11/part-3-unit-testing-with-traits-and-code-coverage-in-visual-studio-2012-using-the-tfs-build-and-the-new-nuget-adapter-approach/ . You have to uninstall the vsix, and then just add that package through the Manage Nuget packages in your solution. If you're uncomfortable about that, please wait until I get a test of the fixed vsix out.

[pvandervelde]

Sweet. I'll try that tomorrow when I'm back at work.

[MaxG117]

I'd prefer to test a vsix package. And thank you!

[pvandervelde]

I now get the following:

------ Discover test started ------
NUnit VS Adapter 2.1.0.8 discovering tests is started
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.EqualsOperatorWithEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.EqualsOperatorWithLeftNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.EqualsOperatorWithNonEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.EqualsOperatorWithRightNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.EquatableEquals
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.EquatableEqualsWithNonEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.HashCodeComparisonForEqualObjects
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.NotEqualsOperatorWithEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.NotEqualsOperatorWithLeftNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.NotEqualsOperatorWithNonEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.NotEqualsOperatorWithRightNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.ObjectEquals
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.ObjectEqualsWithNonEqualObjectOfDifferentType
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.ObjectEqualsWithNonEqualObjectOfSameType
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.EqualityContractVerifierTest.ObjectEqualsWithNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.HashCodeContractVerifierTest.VerifyCollisionProbability
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.BuildDescriptionIdTest.HashCodeContractVerifierTest.VerifyUniformDistribution
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.EqualsOperatorWithEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.EqualsOperatorWithLeftNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.EqualsOperatorWithNonEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.EqualsOperatorWithRightNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.EquatableEquals
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.EquatableEqualsWithNonEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.HashCodeComparisonForEqualObjects
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.NotEqualsOperatorWithEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.NotEqualsOperatorWithLeftNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.NotEqualsOperatorWithNonEqualObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.NotEqualsOperatorWithRightNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.ObjectEquals
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.ObjectEqualsWithNonEqualObjectOfDifferentType
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.ObjectEqualsWithNonEqualObjectOfSameType
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.EqualityContractVerifierTest.ObjectEqualsWithNullObject
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.HashCodeContractVerifierTest.VerifyCollisionProbability
Exception Mono.Cecil.AssemblyResolutionException, Exception converting Vista.DevInfrastructure.Components.Builds.Configuration.Reports.TestToolReportTest.HashCodeContractVerifierTest.VerifyUniformDistribution
NUnit VS Adapter 2.1.0.8 discovering test is finished
========== Discover test finished: 98 found (0:00:02.9951785) ==========
------ Run test started ------
NUnit VS Adapter 2.1.0.8 executing tests is started
Loading tests from C:\vcs\git\devinfrastructure\Components-Builds.Configuration\src\Components.Builds.Configuration.UnitTests\bin\Debug\Vista.DevInfrastructure.Components.Builds.Configuration.UnitTests.dll
Dependent Assembly  of C:\vcs\git\devinfrastructure\Components-Builds.Configuration\src\Components.Builds.Configuration.UnitTests\bin\Debug\Vista.DevInfrastructure.Components.Builds.Configuration.UnitTests.dll not found. Can be ignored if not a NUnit project.
NUnit VS Adapter 2.1.0.8 executing tests is finished
No test is available in C:\vcs\git\devinfrastructure\Components-Builds.Configuration\src\Components.Builds.Configuration.UnitTests\bin\Debug\Vista.DevInfrastructure.Components.Builds.Configuration.UnitTests.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.
========== Run test finished: 0 run (0:00:01.294075) ==========

It finds 98 tests ( think there should be about 150 odd) and doesn't run any of the tests.

[OsirisTerje]

@pvandervelde It seems to be something else going on here, because this error happens at discovery. It absolutely seems related to the Mono.Cecil, but I am not sure what this is. Do you have the possibility to create a small repro file for this issue?

[pvandervelde]

I'll see if I can make something. Will see if I can do it over lunch. Otherwise it'll be end of the day (working day has just started here in New Zealand)

[OsirisTerje]

I also see the executor has a loading issue, so there seems to be some dll it is struggling with. I have checked yet another time now, and all dll's in the package are .net 3.5. If you open your unittests.dll in e.g. Jetbrains dotPeek or similar tool, you can see if the referenced dll's and also their references are really .net 3.5

[pgurenko]

Same issue here. How to rollback the NUnit Test Adapter?

[OsirisTerje]

Here is a vsix with Cecil version 3.5. Can you check @MaxG117 and @pvandervelde ?

NUnitTestAdapter-2.1.1.zip (unzip to get the vsix)

[CharliePoole]

@OsirisTerje Take care with versions and build targets of Cecil as it is bundled with the engine as well as being used by the adapter.

[OsirisTerje]

@CharliePoole Thanks, Charlie. I think I need you to keep an eye on the changes. I'm on thin ice here.

[CharliePoole]

Basically, part of the logic for using Cecil to locate test methods was that we were already using it in the engine, so it's already loaded when the adapter loads the engine. The engine, however, doesn't use the two dlls that support pdbs and mdbs respectively.

If we keep using Cecil, it may end up that having the engine provide source locations for the adapter is a cleaner solution. The gui may want to have the same information.

OTOH, we may want to use some other solution based on the bits that Microsoft has released as Open Source.

[MaxG117]

@OsirisTerje I installed the vsix you provided above and ran a few tests. It works. I ran the same test under Debug to compare the outputs from 2.1.1 and 2.0.0. 2.1.1 showed the following lines that were not observed in 2.0.0:

'vstest.executionengine.x86.clr20.exe' (CLR v2.0.50727: vstest.executionengine.x86.clr20.exe): Loaded 'C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\COMMON7\IDE\COMMONEXTENSIONS\MICROSOFT\TESTWINDOW\Extensions\Microsoft.VisualStudio.TestPlatform.Extensions.dll'. Symbols loaded. Exception thrown: 'System.BadImageFormatException' in mscorlib.dll Additional information: Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.Extensions.GenericTestAdapter' or one of its dependencies. This assembly is built by a runtime newer than the currently loaded runtime and cannot be loaded.

'vstest.executionengine.x86.clr20.exe' (CLR v2.0.50727: vstest.executionengine.x86.clr20.exe): Loaded 'C:\Windows\assembly\GAC_MSIL\Microsoft.VisualStudio.Debugger.Runtime\14.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Debugger.Runtime.dll'.

So there may be another dependent assembly targeting (or "built by") the wrong runtime, even though it doesn't affect the results.

[OsirisTerje]

Basic issue with .net framework 3.5 fixed in release 2.1.1 today.
@pvandervelde : Can you add a new one for your issue? And, I would still appreciate a repro, if you can make it.