VS15.6 Real-Time Test Discovery yields unrunnable NUnit tests

yaakov-h commented 6 years ago

In Visual Studio 15.6 (with real-time test discovery), the Visual Studio Test Explorer shows tests that do not exist, or tests with sources. In both cases, the tests cannot be run via the Test Explorer.

For example, in Visual Studio Enterprise v15.6.3 with NUnit Adapter 3.10.0.21 and NUnit Framework 3.10.1, and the following code that targets .NET Framework:

using NUnit.Framework;
using System.Collections.Generic;

namespace NUnitSetName
{
    public class Class1
    {
        [Test]
        public void SimpleTest() { }

        [TestCase(1)]
        public void TestCase(int i) { }

        [TestCase(1, TestName = "{m} with a 1")]
        [TestCase(1, TestName = "an overriden test name with a 1")]
        public void NamedTestCase(int i) { }

        [TestCaseSource(nameof(Sauce))]
        public void SourcedTestCase(int i) { }

        public static IEnumerable<TestCaseData> Sauce
        {
            get
            {
                yield return new TestCaseData(1).SetName("totally overriden name");
                yield return new TestCaseData(1).SetName("{m} - enhanced name");
            }
        }
    }
}

After opening the solution, the Test Explorer "discovers" tests including {m} as a literal:

After building the solution it runs through the test case sources and shows the proper test names, but does not discard the incorrectly discovered ones that are not actually valid tests:

After running all tests, not all tests were run:

Attempting to select one of the last two tests and running it just yields no result, and a fairly useless messages in the Tests Output window:

[22/03/2018 2:39:32 PM Informational] ------ Run test started ------
[22/03/2018 2:39:32 PM Informational] NUnit Adapter 3.10.0.21: Test execution started
[22/03/2018 2:39:32 PM Informational] Running all tests in C:\Temp\NUnitSetName\NUnitSetName\bin\Debug\NUnitSetName.dll
[22/03/2018 2:39:33 PM Informational] NUnit3TestExecutor converted 6 of 6 NUnit test cases
[22/03/2018 2:39:33 PM Informational] Skipping assembly - no matching test cases found
[22/03/2018 2:39:33 PM Informational] NUnit Adapter 3.10.0.21: Test execution complete
[22/03/2018 2:39:33 PM Warning] No test is available in C:\Temp\NUnitSetName\NUnitSetName\bin\Debug\NUnitSetName.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.
[22/03/2018 2:39:33 PM Informational] ========== Run test finished: 0 run (0:00:01.1529461) ==========

OsirisTerje commented 6 years ago

These are issues with the new source based discovery, and how it discovers tests. We need changes both there and in the adapter to align how tests are named. Until then, this issue is blocked.

yaakov-h commented 6 years ago

Is there an issue for the VS side that I can track?

OsirisTerje commented 6 years ago

Not yet, but I have mailed the PG about this, and expect them to comment here. You can add to Developer Community if you like, the more the better, and link back to this one too.

ManishJayaswal commented 6 years ago

Yes we are aware of this issue with SetName attribute. Anytime the name of a test is changed dynamically ( i.e. requires code to be run) source based discovered copy of that test will not match the test object returned by the NUnit test framework. This happens because VS depends on Fully Qualified Name to dedupe them and if the testcase name has changed then FQN also changes. We are in the process of standardizing the FQN and are in touch with NUnit folks to get that implemented. So I really hope that we can get issues like these resolved soon. @peterwald - FYI

jnm2 commented 6 years ago

I've been running into this as well.

Just so I understand, SetName (or the upcoming SetArgDisplayNames) will still control the display name in test explorer after the FQN work, right?

OsirisTerje commented 6 years ago

Yes, it will. The FQN will be added as an additional element and used by the real time discovery

Barsonax commented 6 years ago

In the process of converting a PCL library to netstandard I have noticed that instead of almost 200 tests I now only have 110 tests of which some are run and some are not. This is due to alot of parameterized tests using SetName I think. This was working fine before I converted to netstandard though so seeing this issue I wonder why it was working before.

If interested this is the repo: https://github.com/Barsonax/Pathfindax. The develop branch is still on PCL and the DotnetStandard has been converted to netstandard. Tested this with Pathfindax and Pathfindax.Test (just unload all other projects).