NUnit failed to load test assembly, test doesn't run

[bording]

When trying to run a test from Visual Studio, I sometimes see the following output, and the test doesn't run:

[1/14/2019 1:57:48 PM Informational] ------ Run test started ------
[1/14/2019 1:57:48 PM Informational] NUnit Adapter 3.12.0.0: Test execution started
[1/14/2019 1:57:48 PM Informational] Running all tests in C:\Code\NServiceBus.Persistence.ServiceFabric\src\Tests\bin\Debug\net452\Tests.dll
[1/14/2019 1:57:48 PM Informational]    NUnit failed to load C:\Code\NServiceBus.Persistence.ServiceFabric\src\Tests\bin\Debug\net452\Tests.dll
[1/14/2019 1:57:48 PM Informational] NUnit Adapter 3.12.0.0: Test execution complete
[1/14/2019 1:57:48 PM Warning] No test matches the given testcase filter `FullyQualifiedName=ApiApproval.Approve` in C:\Code\NServiceBus.Persistence.ServiceFabric\src\Tests\bin\Debug\net452\Tests.dll
[1/14/2019 1:57:48 PM Informational] ========== Run test finished: 0 run (0:00:00.6869999) ==========

This same test runs fine on my CI server.

Is there any way to get more information about why the test assembly could not be loaded?

NUnit 3.11 NUnit3TestAdapter 3.12.0 VS 2017 15.9.5

[OsirisTerje]

NUnit can't find any runnable tests in the assembly, which of course is strange in your case. The adapter states this based on the result it gets back from NUnit itself. You can see this result yourself by enabling the dump feature, and see if there are any clues there to what's going wrong.

To do so, add a runsettings file with enabling the NUnit Dump settings as described here: https://github.com/nunit/docs/wiki/Tips-And-Tricks, using the DumpXmlTestResults

You can also check the results from the DumpXmlTestDiscovery

[bording]

Here's the output I'm seeing:

<?xml version="1.0" encoding="utf-8"?>
<NUnitXml>
<test-run id='2' name='Tests.dll' fullname='C:\Code\NServiceBus.Persistence.ServiceFabric\src\Tests\bin\Debug\net452\Tests.dll' testcasecount='0'>
   <test-suite type='Assembly' id='0-1000' name='Tests.dll' fullname='Tests.dll' runstate='NotRunnable' testcasecount='0'>
      <properties>
         <property name='_SKIPREASON' value='Could not load file or assembly &apos;Tests, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b50674d1e0c6ce54&apos; or one of its dependencies. An attempt was made to load a program with an incorrect format.' />
      </properties>
   </test-suite>
</test-run>
</NUnitXml>

For this specific repo, that made me realize there is an x86/x64 mismatch. Once I fixed that, the test is running properly!

I have seen this in other repos which I don't think would have the same architecture mismatch, but now I have a way to dig into why they are failing.

It would be nice if the default test adapter output could surface more detailed error information somehow without needing a runsettings file, though.

[OsirisTerje]

Good it helped you!

The problem is that we try to keep the output not too crowded, and at the same time there is no other way of injecting in parameters than through the runsettings file. I have been thinking if there is some way to add a default diagnostic runsettings file, and somehow enabling this through the UI. I do agree with you it should be more easy.

About the x86/x64, I have had several similar incidents myself, and we need to find a better way of detecting just this case. One thing we could do is to add the information to the output of which mode (x86 or x64 runner) we're currently running.

[aolszowka]

We have started to encounter this error as well; I feel like something changed with Visual Studio recently that is causing this bug to surface more frequently than it did previously. Our developers see it sporadically and today I was finally able to reproduce it with a small test case.

Environment: Visual Studio 15.9.4 NUnit 3.11.0 NUnit Test Adapter 3.11.2

[2/13/2019 7:47:32 AM Informational] ------ Run test started ------
[2/13/2019 7:47:32 AM Informational] NUnit Adapter 3.11.2.0: Test execution started
[2/13/2019 7:47:32 AM Informational] Running all tests in C:\Users\aceo\source\repos\BadNUnitExamples\BadNUnitExamples\bin\Debug\BadNUnitExamples.dll
[2/13/2019 7:47:33 AM Informational]    NUnit3TestExecutor converted 12 of 12 NUnit test cases
[2/13/2019 7:47:33 AM Informational]    Skipping assembly - no matching test cases found
[2/13/2019 7:47:33 AM Informational] NUnit Adapter 3.11.2.0: Test execution complete
[2/13/2019 7:47:33 AM Warning] No test matches the given testcase filter `FullyQualifiedName=BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty.Divide` in C:\Users\aceo\source\repos\BadNUnitExamples\BadNUnitExamples\bin\Debug\BadNUnitExamples.dll
[2/13/2019 7:47:33 AM Informational] ========== Run test finished: 0 run (0:00:01.5567434) ==========

The Adapter was able to find the tests originally which is confusing why when we go to run the tests it suddenly can no longer find them:

Enabling the DumpXmlTestResults settings as recommended produces this (sorry for the length) I see the class name in here; so I will continue to dig:

<?xml version="1.0" encoding="utf-8"?>
<NUnitXml>
<test-run id='2' name='BadNUnitExamples.dll' fullname='C:\Users\aceo\source\repos\BadNUnitExamples\BadNUnitExamples\bin\Debug\BadNUnitExamples.dll' testcasecount='12'>
   <test-suite type='Assembly' id='0-1029' name='BadNUnitExamples.dll' fullname='BadNUnitExamples.dll' runstate='Runnable' testcasecount='12'>
      <properties>
         <property name='_PID' value='42868' />
         <property name='_APPDOMAIN' value='domain-af528d07-BadNUnitExamples.dll' />
      </properties>
      <test-suite type='TestSuite' id='0-1030' name='BadNUnitExamples' fullname='BadNUnitExamples' runstate='Runnable' testcasecount='12'>
         <test-suite type='TestFixture' id='0-1000' name='NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1' runstate='Runnable' testcasecount='1'>
            <test-suite type='ParameterizedMethod' id='0-1002' name='Multiply' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1.Multiply' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1001' name='DuplicatedTestCase2' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1.DuplicatedTestCase2' methodname='Multiply' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses1' runstate='Runnable' seed='1892311748' />
            </test-suite>
         </test-suite>
         <test-suite type='TestFixture' id='0-1003' name='NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2' runstate='Runnable' testcasecount='1'>
            <test-suite type='ParameterizedMethod' id='0-1005' name='Divide' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2.Divide' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1004' name='DuplicatedTestCase2' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2.DuplicatedTestCase2' methodname='Divide' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsAcrossClasses2' runstate='Runnable' seed='599424301' />
            </test-suite>
         </test-suite>
         <test-suite type='TestFixture' id='0-1006' name='NUnitDuplicatedNameAcrossTwoTestsInSameClass' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass' runstate='Runnable' testcasecount='2'>
            <test-suite type='ParameterizedMethod' id='0-1008' name='Addition' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass.Addition' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1007' name='DuplicateTestCase1' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass.DuplicateTestCase1' methodname='Addition' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass' runstate='Runnable' seed='1486660708' />
            </test-suite>
            <test-suite type='ParameterizedMethod' id='0-1010' name='Subtraction' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass.Subtraction' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1009' name='DuplicateTestCase1' fullname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass.DuplicateTestCase1' methodname='Subtraction' classname='BadNUnitExamples.NUnitDuplicatedNameAcrossTwoTestsInSameClass' runstate='Runnable' seed='526825222' />
            </test-suite>
         </test-suite>
         <test-suite type='TestFixture' id='0-1011' name='NUnitDuplicatedNamesAttribute' fullname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' classname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' runstate='Runnable' testcasecount='3'>
            <test-suite type='ParameterizedMethod' id='0-1014' name='Mulitply' fullname='BadNUnitExamples.NUnitDuplicatedNamesAttribute.Mulitply' classname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' runstate='Runnable' testcasecount='2'>
               <test-case id='0-1012' name='Mulitply Test1' fullname='BadNUnitExamples.NUnitDuplicatedNamesAttribute.Mulitply Test1' methodname='Mulitply' classname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' runstate='Runnable' seed='1909484835' />
               <test-case id='0-1013' name='Mulitply Test1' fullname='BadNUnitExamples.NUnitDuplicatedNamesAttribute.Mulitply Test1' methodname='Mulitply' classname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' runstate='Runnable' seed='2134959053' />
            </test-suite>
            <test-suite type='ParameterizedMethod' id='0-1016' name='Mulitply2' fullname='BadNUnitExamples.NUnitDuplicatedNamesAttribute.Mulitply2' classname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1015' name='Mulitply2(2,1)' fullname='BadNUnitExamples.NUnitDuplicatedNamesAttribute.Mulitply2(2,1)' methodname='Mulitply2' classname='BadNUnitExamples.NUnitDuplicatedNamesAttribute' runstate='Runnable' seed='1858079574' />
            </test-suite>
         </test-suite>
         <test-suite type='TestFixture' id='0-1017' name='NUnitDuplicateNamesIEnumerableProperty' fullname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty' classname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty' runstate='Runnable' testcasecount='2'>
            <test-suite type='ParameterizedMethod' id='0-1020' name='Divide' fullname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty.Divide' classname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty' runstate='Runnable' testcasecount='2'>
               <test-case id='0-1018' name='Divide Test2' fullname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty.Divide Test2' methodname='Divide' classname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty' runstate='Runnable' seed='481009269' />
               <test-case id='0-1019' name='Divide Test2' fullname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty.Divide Test2' methodname='Divide' classname='BadNUnitExamples.NUnitDuplicateNamesIEnumerableProperty' runstate='Runnable' seed='2080622784' />
            </test-suite>
         </test-suite>
         <test-suite type='TestFixture' id='0-1021' name='NUnitExpectedExceptionExamples' fullname='BadNUnitExamples.NUnitExpectedExceptionExamples' classname='BadNUnitExamples.NUnitExpectedExceptionExamples' runstate='Runnable' testcasecount='1'>
            <test-case id='0-1022' name='TestingTheFunctionIReallyWannaTest' fullname='BadNUnitExamples.NUnitExpectedExceptionExamples.TestingTheFunctionIReallyWannaTest' methodname='TestingTheFunctionIReallyWannaTest' classname='BadNUnitExamples.NUnitExpectedExceptionExamples' runstate='Runnable' seed='1138631328' />
         </test-suite>
         <test-suite type='TestFixture' id='0-1023' name='NUnitTestCaseDataResult' fullname='BadNUnitExamples.NUnitTestCaseDataResult' classname='BadNUnitExamples.NUnitTestCaseDataResult' runstate='Runnable' testcasecount='1'>
            <test-suite type='ParameterizedMethod' id='0-1025' name='Subtraction' fullname='BadNUnitExamples.NUnitTestCaseDataResult.Subtraction' classname='BadNUnitExamples.NUnitTestCaseDataResult' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1024' name='Test' fullname='BadNUnitExamples.NUnitTestCaseDataResult.Test' methodname='Subtraction' classname='BadNUnitExamples.NUnitTestCaseDataResult' runstate='Runnable' seed='1418979146' />
            </test-suite>
         </test-suite>
         <test-suite type='TestFixture' id='0-1026' name='NUnitTestCaseDataStatic' fullname='BadNUnitExamples.NUnitTestCaseDataStatic' classname='BadNUnitExamples.NUnitTestCaseDataStatic' runstate='Runnable' testcasecount='1'>
            <test-suite type='ParameterizedMethod' id='0-1028' name='Addition' fullname='BadNUnitExamples.NUnitTestCaseDataStatic.Addition' classname='BadNUnitExamples.NUnitTestCaseDataStatic' runstate='Runnable' testcasecount='1'>
               <test-case id='0-1027' name='TestAddition' fullname='BadNUnitExamples.NUnitTestCaseDataStatic.TestAddition' methodname='Addition' classname='BadNUnitExamples.NUnitTestCaseDataStatic' runstate='Runnable' seed='1439801385' />
            </test-suite>
         </test-suite>
      </test-suite>
   </test-suite>
</test-run>
</NUnitXml>

I will start by first upgrading to the new test adapter to see if anything gets magically fixed. This did work at one time without issue.

[aolszowka]

So looking at this specific case closer: This is using a TestCaseSource to generate test cases; none of those additional tests are showing up. I wonder if previously it would run one of the test cases. I am checking with our developers who encountered this issue in production code if the tests they are seeing the issue with are using a similar pattern.

[OsirisTerje]

@aolszowka Can you create a workable small project which reproduces this case, as I see you mention you managed to extract a small test case above ?

[aolszowka]

I think I am discovering another bug or another issue that seems to have the same symptoms; below is the breakdown:

namespace NUnitVSRunnerFailsOnTestCaseSource
{
    using System.Collections.Generic;
    using NUnit.Framework;

    [TestFixture]
    class TestCaseIEnumerableDuplicateNames
    {
        static IEnumerable<TestCaseData> AddTests
        {
            get
            {
                yield return new TestCaseData(0, 0).Returns(0).SetName("Test");
                yield return new TestCaseData(1, 1).Returns(2).SetName("Test");
            }
        }

        [TestCaseSource(nameof(AddTests))]
        public int Add(int a, int b)
        {
            int actual = a + b;
            return actual;
        }
    }
}

If you attempt to run this fixture you get this in the runner:

Note how the "Add" test shows as (!) and you are unable to run it (I will call this a "Ghost" test). This makes sense as there is nothing to run at the "Add" method; all of the logic is provided by the AddTests Method. I feel like previously you were able to run the "Add" test and it would run all of the testcase source tests but maybe I am just crazy.

Also notice that while the names are duplicated this behavior seems to be "better" in that it will indicate additional runs now like so:

This Ghost Test does not occur if you do not specify a name for example this code snippet will produce no "Ghosts":

namespace NUnitVSRunnerFailsOnTestCaseSource
{
    using System.Collections.Generic;
    using NUnit.Framework;

    [TestFixture]
    public class TestCaseIEnumerable
    {
        static IEnumerable<TestCaseData> AddTests
        {
            get
            {
                yield return new TestCaseData(0, 0).Returns(0);
                yield return new TestCaseData(1, 1).Returns(2);
            }
        }

        [TestCaseSource(nameof(AddTests))]
        public int Add(int a, int b)
        {
            int actual = a + b;
            return actual;
        }
    }

}

If you set just a single name like so:

namespace NUnitVSRunnerFailsOnTestCaseSource
{
    using System.Collections.Generic;
    using NUnit.Framework;

    [TestFixture]
    public class TestCaseIEnumerable
    {
        static IEnumerable<TestCaseData> AddTests
        {
            get
            {
                yield return new TestCaseData(0, 0).Returns(0).SetName("A");
                yield return new TestCaseData(1, 1).Returns(2);
            }
        }

        [TestCaseSource(nameof(AddTests))]
        public int Add(int a, int b)
        {
            int actual = a + b;
            return actual;
        }
    }

}

You still will not see a Ghost:

But if you set both names you will see the ghost; even if they are unique names like so:

namespace NUnitVSRunnerFailsOnTestCaseSource
{
    using System.Collections.Generic;
    using NUnit.Framework;

    [TestFixture]
    public class TestCaseIEnumerable
    {
        static IEnumerable<TestCaseData> AddTests
        {
            get
            {
                yield return new TestCaseData(0, 0).Returns(0).SetName("A");
                yield return new TestCaseData(1, 1).Returns(2).SetName("B");
            }
        }

        [TestCaseSource(nameof(AddTests))]
        public int Add(int a, int b)
        {
            int actual = a + b;
            return actual;
        }
    }

}

Attempting to run this "Ghost" test results in the error I was seeing previously:

[2/13/2019 12:28:56 PM Informational] ------ Run test started ------
[2/13/2019 12:28:56 PM Informational] NUnit Adapter 3.12.0.0: Test execution started
[2/13/2019 12:28:56 PM Informational] Running all tests in R:\NUnitRepo\NUnitVSRunnerFailsOnTestCaseSource\NUnitVSRunnerFailsOnTestCaseSource\bin\Debug\NUnitVSRunnerFailsOnTestCaseSource.dll
[2/13/2019 12:28:56 PM Informational]    NUnit3TestExecutor converted 4 of 4 NUnit test cases
[2/13/2019 12:28:56 PM Informational]    Skipping assembly - no matching test cases found
[2/13/2019 12:28:56 PM Informational] NUnit Adapter 3.12.0.0: Test execution complete
[2/13/2019 12:28:56 PM Warning] No test matches the given testcase filter `FullyQualifiedName=NUnitVSRunnerFailsOnTestCaseSource.TestCaseIEnumerable.Add` in R:\NUnitRepo\NUnitVSRunnerFailsOnTestCaseSource\NUnitVSRunnerFailsOnTestCaseSource\bin\Debug\NUnitVSRunnerFailsOnTestCaseSource.dll
[2/13/2019 12:28:56 PM Informational] ========== Run test finished: 0 run (0:00:00.6330908) ==========

Is there any way for us to remove these Ghosts? What would the expected behavior be?

[OsirisTerje]

@aolszowka I assume you have the Source Based Discovery (SDB) turned on? The SDB cannot see what a testcasesource will give out, so it will just show the Add method. Once you run this, the real tests will pop up, and the Add should disappear, or show as a parent node (Depends on how you choose to view this).

[aolszowka]

We have Source Based Discovery Turned on; I see that when I turn it off I do not get the "ghosting".

Based on that I am assuming that is expected then?

[OsirisTerje]

@aolszowka Yes it is, and I think the MS PG is aware of this issue, it is really on their side, and I assume they eventually will add some information there to point out that it is a temporary node.

[gabbsmo]

In my case it was Visual Studio that added and changed some dependentAssembly elements in the App.config of the test project. Undoing these changes made the tests run again.

EDIT: I could repro this by updating NUnit from Nuget. This would add System.Net.Http as a dependentAssempy. Remove this element from app.config makes the tests run again.

[manacu]

I was having the same issue VS 2019. After adding the extension (https://marketplace.visualstudio.com/items?itemName=NUnitDevelopers.NUnit3TestAdapter) then it worked