Handling Duplicate Test IDs in TRX Files

[rif281]

Hello,

I encountered an issue when using trxlog2html where the tool fails if there are tests with the same ID within the same TRX file. The error message is:

System.ArgumentException: An item with the same key has already been added.

Suggestion: To handle this issue, the tool could modify duplicate testId and testName values by appending a unique suffix, ensuring all IDs and names are unique within the TRX file.

Thank you very much.

Best regards, Reef Ashkenazi.

[HikosakaRyo]

Hello Reef Ashkenazi,

Thank you for using my tool. I am happy to assist you and will do my best to address the issue!

The testId in UnitTestResult is used to link UnitTestResult with UnitTest, so if there are duplicate testIds, it might be possible to link them using testId + testName as the key.

However, in my environment, I have not been able to reproduce the issue of duplicate testIds, which is a concern. The TRX files output by vstest.console.exe contain testIds that look like UUIDs, making the likelihood of duplication extremely low.

For example:

<UnitTestResult executionId="0293f70a-ca38-4b66-8440-ba1897dbc384" testId="336ff03b-77e0-b087-2767-1650443bfb18" testName="Foo() returns &quot;foo&quot;" computerName="xxxx" duration="00:00:00.0000995" startTime="2024-06-01T06:06:19.8483691+09:00" endTime="2024-06-01T06:06:19.8483695+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="0293f70a-ca38-4b66-8440-ba1897dbc384" />

Could you please share how you are generating the TRX files in your case and what kind of testIds are being produced?

Thank you and best regards.

[rif281]

Hello

Thanks for the reply. We're using the xUnit framework with Theory attributes, which allows us to test multiple inputs against the same test logic. This results in duplicate test IDs in our TRX files. We create the TRX file using a task in the Azure DevOps pipeline with the following configuration:

  - task: DotNetCoreCLI@2
    displayName: '.Net Test'
    inputs:
      command: 'test'
      projects: ${{ parameters.projects }}
      arguments: >-
        --no-restore
        --no-build
        ${{ parameters.runtime }}
        /p:Configuration='Release'
        /p:Version=$(Build.BuildNumber)
        --settings:$(CodeCoveragePath)
        --collect:"XPlat code coverage"
        --verbosity 'Normal'
        --logger trx
        --results-directory $(Common.TestResultsDirectory)

As a potential solution, we suggest using executionId instead of testId to differentiate between the tests. Any insights or suggestions you have to address this would be greatly appreciated.

Thank you!

[HikosakaRyo]

Hello.

Thank you for the detailed information! I attempted to reproduce the issue in my environment by writing test cases using the Theory attribute, but I was unable to replicate the problem of duplicate test IDs.

Here is the test code I used:

[Theory]
[InlineData(0)]
[InlineData(1)]
[InlineData(2)]
public void BazTest(int value)
{
  // Test logic here
}

And here is the resulting TRX file snippet:

<Results>
  <UnitTestResult executionId="a1bb2af3-5345-48b1-8c7a-b05832675380" testId="5b1f47a5-4cb8-ba67-f7e4-715d4e4c823a" testName="UnitTestSample.Tests.FooBarBazTest.BazTest(value: 1)" computerName="xxx" duration="00:00:00.0000037" startTime="2024-06-03T18:06:26.0076845+09:00" endTime="2024-06-03T18:06:26.0076846+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="a1bb2af3-5345-48b1-8c7a-b05832675380" />
  <UnitTestResult executionId="50123cfd-ebc7-4489-b4a3-c54bfdb5be1b" testId="1e4ede35-b087-58dd-3de2-cba0fc882d6b" testName="UnitTestSample.Tests.FooBarBazTest.BazTest(value: 2)" computerName="xxx" duration="00:00:00.0038375" startTime="2024-06-03T18:06:26.0045327+09:00" endTime="2024-06-03T18:06:26.0045837+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="50123cfd-ebc7-4489-b4a3-c54bfdb5be1b" />
  <UnitTestResult executionId="d6b9e83c-09fe-4d0e-ac1f-feb3f8221131" testId="44f5a3ee-8726-0292-e8c6-e1aa3ff37f8b" testName="UnitTestSample.Tests.FooBarBazTest.BazTest(value: 0)" computerName="xxx" duration="00:00:00.0000193" startTime="2024-06-03T18:06:26.0076637+09:00" endTime="2024-06-03T18:06:26.0076640+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="d6b9e83c-09fe-4d0e-ac1f-feb3f8221131" />
  ...
</Results>

Do you notice anything unusual in this test code? One thing to note is that I am using the vstest.console.exe command to run the tests, not the test command in the Azure DevOps pipeline. This difference might be causing the variation in behavior regarding the test IDs.

Nevertheless, using executionId instead of testId as the key for linking might be worth trying. If I push the fix to a separate branch, would you be able to test it in your environment? Alternatively, if it's possible for you to share the problematic TRX file, that could also help in diagnosing the issue further.

Best regards,

[rif281]

Hi

I noticed that you are using InlineData in your tests. In our case, we are using MemberData, which might behave differently and could be contributing to the issue.

Yes, I will be able to test the changes in my environment.

Unfortunately, I can't share the TRX file because it contains private data.

Thank you for your understanding and assistance.

[HikosakaRyo]

Hello

I have created a branch, "fix_issue_2" and pushed the changes to use executionId instead of testId. Could you please check if it works in your environment? Unfortunately, I was unable to reproduce the duplicate testId issue with MemberData tests in my environment (please see the details below). Therefore, I have manually created a TRX file with duplicate testIds to perform minimal testing in my environment.

Here is the test code I wrote:

public static IEnumerable<object[]> Data => new List<object[]>
{
    new object[] { 1, 2, "baz_1_2" },
    new object[] { 3, 4, "baz_3_4" },
    new object[] { 5, 6, "baz_5_6" }
};

[Theory]
[MemberData(nameof(Data))]
public void BazTest(int val1, int val2, string expected)
{
  // Test logic here
}

And here is the resulting TRX file snippet:

<Results>
  <UnitTestResult executionId="5d6c753c-2d6a-49b6-8036-5e5af2779861" testId="db80d851-ca14-7b19-3891-7771e1323bf3" testName="UnitTestSample.Tests.FooBarBazTest.BazTest(val1: 5, val2: 6, expected: &quot;baz_5_6&quot;)" computerName="xxx" duration="00:00:00.0000054" startTime="2024-06-04T10:21:46.8690775+09:00" endTime="2024-06-04T10:21:46.8690776+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="5d6c753c-2d6a-49b6-8036-5e5af2779861" />
  <UnitTestResult executionId="67bb4118-5472-4a5b-b906-1a4717b891cf" testId="96cb3785-ea84-5d8f-80ad-0e90507ab594" testName="UnitTestSample.Tests.FooBarBazTest.BazTest(val1: 3, val2: 4, expected: &quot;baz_3_4&quot;)" computerName="xxx" duration="00:00:00.0000265" startTime="2024-06-04T10:21:46.8690515+09:00" endTime="2024-06-04T10:21:46.8690518+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="67bb4118-5472-4a5b-b906-1a4717b891cf" />
  <UnitTestResult executionId="777b99e5-c68b-4298-b52e-90f877a73a05" testId="698ddf1c-fe49-79e0-ce5c-f333c09d1b72" testName="UnitTestSample.Tests.FooBarBazTest.BazTest(val1: 1, val2: 2, expected: &quot;baz_1_2&quot;)" computerName="xxx" duration="00:00:00.0024175" startTime="2024-06-04T10:21:46.8684092+09:00" endTime="2024-06-04T10:21:46.8684098+09:00" testType="13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b" outcome="Passed" testListId="8c84fa94-04c1-424b-9868-57a2d4851a1d" relativeResultsDirectory="777b99e5-c68b-4298-b52e-90f877a73a05" />
</Results>

Thank you for your cooperation.

Best regards,

[inbarbarkai]

Here is a repro of the issue:

using Xunit;

namespace trxtest
{
    public class TestClass
    {
        [MemberData(nameof(TestData))]
        [Theory]
        public void Test(TestInput input)
        {
            Assert.Equal(input.Value, input.Value2);
        }

        public static TheoryData<TestInput> TestData()
        {
            return new TheoryData<TestInput>    {
                new TestInput { Value= "some" },
                new TestInput { Value= "foo" }
            };
        }
    }

    public class TestInput
    {
        public string Value { get; set; }
        public string Value2 { get; set; }
    }
}

I guess it is because the class is not serializable.

[HikosakaRyo]

Hello, Inbar Barkai.

Thank you for your code! I was able to reproduce the issue using your example. It also became clear that the code I pushed was insufficient to handle such cases. (Multiple UnitTestResult elements were linked to a single UnitTest element by testId, and the name of the UnitTest element was not created for each input.)

Your suggestion addresses both of these issues and ensures correct operation even in cases of duplicate testId.

@rif281 If there are no issues, I am considering merging @inbarbarkai's code. Do you think this code will resolve your issue?

[inbarbarkai]

Hi @HikosakaRyo, I am a colleague of @rif281. I tested the solution on our problematic trx file.

[HikosakaRyo]

@rif281, @inbarbarkai I have merged the code you provided. Thank you again!