microsoft / testfx

MSTest framework and adapter
MIT License
702 stars 251 forks source link

TRX report: ResultFiles are attached to every datarow #3277

Closed baumheld closed 1 month ago

baumheld commented 1 month ago

Describe the bug

When using Datarows and ResultFiles together, the ResultFiles of every datarow are attached to all other datarows too, not just the current one.

Steps To Reproduce

  1. Checkout this repo https://github.com/baumheld/mstest_multiple_ResultFiles/
  2. Execute RunTests.ps1 in Powershell (Right-click it).
    This will run a single test with 5 datarows. (see below)
    Each datarow will attach one ResultFile
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace TestProject;

[TestClass]
public class TestClass
{
    public TestContext TestContext { get; set; }

    [TestMethod]
    [DataRow("DataRow1")]
    [DataRow("DataRow2")]
    [DataRow("DataRow3")]
    [DataRow("DataRow4")]
    [DataRow("DataRow5")]
    public void TestMethodAsync(string datarowName)
    {
        TestContext.AddResultFile(datarowName);
    }
}
  1. Open newly created TRX file under ..\tests\logs\ with an editor.
    Each test (one test for each datarow) now contains 5 resultfiles.
    Not only does one test contain their own ResultFile, but also all ResultFiles of the other datarows.

Expected behavior

[...]
    <UnitTestResult testName="TestMethodAsync (DataRow1)" >
      <ResultFiles>
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow1" />
      </ResultFiles>
    </UnitTestResult>
    <UnitTestResult testName="TestMethodAsync (DataRow2)" >
      <ResultFiles>
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow2" />
      </ResultFiles>
    </UnitTestResult>
[...]

Actual behavior

[...]
    <UnitTestResult testName="TestMethodAsync (DataRow1)" >
      <ResultFiles>
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow1" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow2" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow3" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow4" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow5" />
      </ResultFiles>
    </UnitTestResult>
    <UnitTestResult testName="TestMethodAsync (DataRow2)" >
      <ResultFiles>
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow1" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow2" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow3" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow4" />
        <ResultFile path="T:\src\mstest_multiple_ResultFiles\bin\Debug\net8.0\DataRow5" />
      </ResultFiles>
    </UnitTestResult>
[...]

Additional context

I'm quite sure this bug was introduced with the MSTest SDK.
I have older .trx-files generated with MSTest meta nuget package, which have just one ResultFile per datarow.

Evangelink commented 1 month ago

Hi @baumheld!

Thanks for the bug report! Is this issue causing severe impact in your testing process (just trying to understand if it can be fixed as part of MSTest 3.6 or if we need to ship a bug fix version for it).

baumheld commented 1 month ago

Is there a roadmap or ETA for 3.6?

To me, it's a minor bug. Can wait until 3.6

Evangelink commented 1 month ago

We just started the iteration so I would guess sometime around mid-to-end August. If we have some other issues, I'll definitely make sure to backport the fix for this one too.

Evangelink commented 1 month ago

I have done a first investigation and the code is not problematic, the problem is coming from #3247 where we end up with all data nodes having the same UID.