search

microsoft / testfx

MSTest framework and adapter
MIT License
697 stars 250 forks source link

Duplicated UIDs for tests with data rows #3247

Closed jakubch1 closed 1 month ago

jakubch1 commented 1 month ago

Describe the bug

UIDs for testing platform protocol are duplicated for tests with data rows. It is causing in VS Preview that if you click to run 1 test with data row => all tests are executed. We are getting in repo:

Duplicated tests found for module 'D:\a_work\1\s\artifacts\bin\Microsoft.CodeCoverage.VisualStudio.Extension.Tests\Debug\net481\Microsoft.CodeCoverage.VisualStudio.Extension.Tests.exe', total duplicate tests: 37 Duplicated tests found for module 'D:\a_work\1\s\artifacts\bin\Microsoft.CodeCoverage.VisualStudio.Window.Tests\Debug\net481\Microsoft.CodeCoverage.VisualStudio.Window.Tests.exe', total duplicate tests: 22 Duplicated tests found for module 'D:\a_work\1\s\artifacts\bin\Microsoft.VisualStudio.CoverageServices.Tests\Debug\net9.0\Microsoft.VisualStudio.CoverageServices.Tests.dll', total duplicate tests: 2 Duplicated tests found for module 'D:\a_work\1\s\artifacts\bin\Microsoft.VisualStudio.TraceDataCollector.Tests\Debug\net481\Microsoft.VisualStudio.TraceDataCollector.Tests.exe', total duplicate tests: 18 Duplicated tests found for module 'D:\a_work\1\s\artifacts\bin\Microsoft.VisualStudio.TraceDataCollector.Tests\Debug\net9.0\Microsoft.VisualStudio.TraceDataCollector.Tests.dll', total duplicate tests: 18

Steps To Reproduce

Create MSTest runner project. Add data rows into test. Execute 1 of them in VS Preview with testing platform protocol enabled.

Expected behavior

Only 1 test is executed (1 data row)

Actual behavior

All tests are executed.

Additional context

Evangelink commented 1 month ago

@jakubch1 @MarcoRossignoli I have investigated and this is not a regression. I have also confirmed that the FQN won't solve this issue as it's only giving a more precise definition of the test method (with parameter types) but not providing any info about arguments.

I don't see a good way to fix that other than:

  1. Changing design so that the data includes some kind of identifier that we could use (similarly to what is done in TATF). But this is a breaking change and would for some time work only for MSTest

  2. Add a new property with some kind of a index info but that assumes that the data is always found at the same index for multiple executions.

Evangelink commented 1 month ago

We did a few more investigations and tests and it seems the requirement from VS Test Explorer is no longer needed so we can change the TestNodeUID to be using the GUID from TestCase.