Open denvermadsen opened 4 years ago
@denvermadsen Thanks for the issue and repro!
This is part of the issues we have with FQNs. I'll mark it as one more of those.
The workaround for now is to add a runsettings file with the following two options in the NUnit section:
<UseParentFQNForParametrizedTests>true</UseParentFQNForParametrizedTests>
<UseNUnitIdforTestCaseId>true</UseNUnitIdforTestCaseId>
It seems I've faced the similar issue when using IEnumerable<record_type>
as TestCaseSource
input.
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.2.0" />
<PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
<PackageReference Include="coverlet.collector" Version="3.1.2">
public record SampleTestCase(int? A, int? B, int ExpectedResult, string Because = null);
public class SampleTestCases
{
public static IEnumerable<SampleTestCase> GetTestCases(bool includeNullValuesForA)
{
if (includeNullValuesForA)
{
yield return new SampleTestCase(null, null, 0);
yield return new SampleTestCase(null, 2, -1, "null value should have precedence over others");
}
yield return new SampleTestCase(1, null, 1, "null value should have precedence over others");
yield return new SampleTestCase(1, 2, -1);
yield return new SampleTestCase(4, 2, 1);
yield return new SampleTestCase(4, 4, 0);
}
}
[TestFixture]
public class SampleComparerTests
{
[TestCaseSource(
typeof(SampleTestCases),
nameof(SampleTestCases.GetTestCases),
new object[] { true })]
public void SampleComparableTestA(SampleTestCase testCase)
{
testObj.Compare(testCase.A, testCase.B).Should().Be(testCase.ExpectedResult, testCase.Because);
}
[TestCaseSource(
typeof(SampleTestCases),
nameof(SampleTestCases.GetTestCases),
new object[] { false })]
public void SampleComparableTestB(SampleTestCase testCase)
{
testObj.Compare(testCase.A, testCase.B).Should().Be(testCase.ExpectedResult, testCase.Because);
}
}
In case when running all tests only first yield is running the test, others are skipped.
Moreover, when test case set is used in multiple test methods - it will run only the first one, others will be skipped entirely.
In the example above only one yield will be run for SampleComparableTestA
and no test cases will be run for SampleComparableTestB
at all.
In the test output window I can see that only 1 test is discovered by the execution engine, although Visual Studio UI shows all 6 cases available.
NUnit3TestExecutor discovered 1 of 1 NUnit test cases using Current Discovery mode, Non-Explicit run
Meanwhile, all test cases are yielded, discovered and run in Live Unit Testing Mode, without issues.
Suggested workaround with .runsettings parameters for NUnit didn't work for me.
Environment:
Summary: Unit tests parameterized with an IEnumerable of tuples (via TestCaseSource) is not run with Test Explorer's Run All button.
Expected Behavior: All test cases are enumerated in Test Explorer with Fail or Pass statuses upon pressing Run All. In sample provided, it is expected that the test case will fail with a red X badge in Test Explorer.
Current Behavior: Test Cases are not run, status for tests are unfilled blue badges. Outputs the following error:
========== Test discovery finished: 1 Tests found in 686 ms ========== ---------- Starting test run ---------- NUnit Adapter 3.17.0.0: Test execution started An exception occurred while invoking executor 'executor://nunit3testexecutor/': Incorrect format for TestCaseFilter Error: Missing '('. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed. ========== Test run finished: 0 Tests run in 460.3 ms (0 Passed, 0 Failed, 0 Skipped) ==========
Additional Notes: A workaround is observed: test runs if using
TestCaseData.SetArgDisplayNames()
Repro steps: Either: open solution in Sample.zip and run tests in Test Explorer OR:
IEnumerable<(object, object)>
and method attribute[TestCaseSource(...)]
TestCaseSource
, implementIEnumerable
interface and return a test case with tuple, e.g.yield return new TestCaseData(new[] {(new object(), new object())});