VSTest generates different TestIDs than Visual Studio and MSTest

[TroyWalshProf]

Description

The TRX file generated by VSTest comes up with a different TestId than MSTest.

This is a problem because this ID is used by TFS to link an automated test to a test case work item. The result is tools like TCM (https://msdn.microsoft.com/en-us/library/ff942470.aspx) cannot upload results (TRX files) into MTM (https://msdn.microsoft.com/en-us/library/jj635157.aspx).

How they differ: MSTest generates the TestID based off the fully qualified name VSTest generates the TestID based off the executor URI, plus the source, plus the fully qualified name Specific line of code that is different: src/Microsoft.TestPlatform.ObjectModel/TestCase.cs#L232

Steps to reproduce

1. Create a simple unit test dll

2. Us MSTest to run the test and create a log file "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\MSTest.exe" /testcontainer:"C:\TEST.dll" /resultsfile:testResults.trx

3. Us VSTest to run the test and create a log file "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "C:\TEST.dll" /logger:trx

4. Compare the resulting TRX files and note that the TestIDs are different

Expected behavior

The TestIDs would match up or we would have a switch or something to generate the MSTest style TestIDs

Actual behavior

The only way I know of to generate these old IDs is to use VSTest with a TestSettings file, but that has other problems.

Environment

vstest.console - All versions This issue has existed for several years https://social.msdn.microsoft.com/Forums/vstudio/en-US/e025b43a-edaa-4ab0-aa07-41ef548f6457/vstestconsoleexe-generates-different-testid-than-visual-studio-and-mstest?forum=vsunittest#98258dfc-b736-4a3d-8b00-9730cb503ed9

[pvlakshm]

vstest does not guarantee that the test IDs will match up with what was generated using mstest.exe.

[TroyWalshProf]

@pvlakshm Was hoping for a little more than a "because I said so" answer here.

This fundamentally changes what TestID is.
When you link a unit test to a test case in Visual Studio it generates a TestID based off the tests fully qualified name. With MSTest the TestID in the TRX file is the same as the TestID in TFS. Tools like TCM depend on this linkage.

Here is why they are different: How MSTest (and VSTest in legacy mode) calculates the ID:

   EqtHash.GuidFromString(this.FullyQualifiedName);

How VSTest calulates the ID:

 string testcaseFullName = this.ExecutorUri + source + this.FullyQualifiedName;
  EqtHash.GuidFromString(testcaseFullName);

I really don’t understand why ExecutorUri and source got added to the TestID. It changes what the TestID represents and given the code comments it doesn’t seem the new implementation really does whatever it was designed to do in the first place.

At the very least could you shed some light onto why changing the TestID was a good thing?

[pvlakshm]

My apologies.

A test name of the form Namespace.Classname.TestMethod can exist in multiple test sources. Furthermore, the pluggable adapter model that vstest supports allows such a test to be in a variety of test frameworks. The adapter in use is identified with the ExecutorUri. Taken together thus, they help identify a test.

Hope this clarifies.