nunit / nunit3-vs-adapter

NUnit 3.0 Visual Studio test adapter for use under VS 2012 or later
https://nunit.org
MIT License
202 stars 104 forks source link

TestAdapter and overloaded parametrized test methods creates warnings and marks wrong status #1166

Closed jgraber closed 1 month ago

jgraber commented 3 months ago

Problem We have two parameterized test methods with the same name in the same test file, but with a different method signature. When we run the tests with the ReSharper Test runner, it marks the tests from the second method as inconclusive and puts the parameters for the second method as pending under the first method.

TestRunner_Bug

2024.04.15 10:40:15.469 WARN Run: e6188ca1-157d-491e-bbf6-44a9ab53679c - Element <Demo.DemoProblemTestRunner.TestMethod(1,True)> doesn't exist in run 2024.04.15 10:40:15.469 WARN Run: e6188ca1-157d-491e-bbf6-44a9ab53679c - Element <Demo.DemoProblemTestRunner.TestMethod(2,False)> doesn't exist in run 2024.04.15 10:40:15.496 WARN Element <Demo.DemoProblemTestRunner.TestMethod(2,False)> was left Pending after its run completion. 2024.04.15 10:40:15.496 WARN Element <Demo.DemoProblemTestRunner.TestMethod(1,True)> was left Pending after its run completion.

Expected behaviour We would expect that both test methods run and that all tests get the correct status.

Example

using NUnit.Framework;
using NUnit.Framework.Legacy;

namespace DemoRSharperTestrunProblem
{
    [TestFixture]
    public class DemoProblemTestRunner
    {
        [Test]
        [TestCase(Language.DE, true)]
        [TestCase(Language.EN, false)]
        public void TestMethod(Language language, bool expected)
        {
            var logic = new BizLogic();

            var result = logic.Method(language);
            ClassicAssert.AreEqual(expected, result);
        }

        [Test]
        [TestCase(1, true)]
        [TestCase(2, false)]
        public void TestMethod(int languageCode, bool expected)
        {
            var logic = new BizLogic();

            var result = logic.Method((Language)languageCode);

            ClassicAssert.AreEqual(expected, result);
        }
    }

    public class BizLogic
    {
        public bool Method(Language input)
        {
            if ((int)input == 1)
            {
                return true;
            }

            return false;
        }
    }

    public enum Language
    {
        DE = 1,
        EN = 2
    }
}
OsirisTerje commented 1 month ago

Sorry for noticing this so late.

As this error happens with Resharper, it should be posted to Jetbrains support.
Resharper doesn't use the adapter, they connect their own runner directly to the NUnit.Engine. But, they do receive the exact same results from the framework, so this looks like a bug in their runner or test display.

PS: The repro code above has been uploaded to the adapter issues repo.