yirkha
commented
1 year ago This is caused by a VSTest task upgrade from 2.205.0 to 2.210.0, done in #16853. It broke handling of tests with a different/custom display name and fully qualified name.
It will not help to update or change the versions of Visual Studio, testing framework or build agents. The VSTest@2/3 tasks need to be fixed (they reference a newer dependency which broke backwards compatibility).
When tests are simply run without extra options, VSTest.Console.exe is executed directly, the test results are published by the agent to the pipeline and that's it.
However, if retries are enabled, a closed source "DtaExecutionHost" component is put in between. It runs VSTest.Console.exe potentially multiple times, parses the output .trx files itself, publishes that to the pipeline, determines the failing ones back through some REST API, handles the reruns, and so on.
This DtaExecutionHost got upgraded from 19.204.32418.1 to 19.210.32906.4 during the upgrade of the VSTest ADO task. The way how a dependency "Microsoft.TeamFoundation.TestClient.PublishTestResults.dll" inside the DtaExecutionHost parses the .trx files changed, and it broke the test reruns for customers.
For example, for a parametrized test case with the following definition
[TestMethod]
[DataTestMethod]
[DataRow("One")]
[DataRow("Two")]
public void TestMethod(string Parameter)The .trx file will have something like this:
<TestDefinitions>
<UnitTest name="TestMethod (One)"
storage="c:\whatever\bin\debug\net6.0\testproject.dll"
id="df932f1e-0bdf-4fd8-b4e0-1d20f4f13e98">
<Execution id="101c7677-781f-43dd-9240-442df2afb88d" />
<TestMethod codeBase="C:\Whatever\bin\Debug\net6.0\TestProject.dll"
adapterTypeName="executor://mstestadapter/v2"
className="Example.Project.Tests.TestClass"
name="TestMethod" />
</UnitTest>Note the <UnitTest name="TestMethod (One)" ... has spaces and parentheses.
But <TestMethod className="className="Example.Project.Tests.TestClass" name="TestMethod" ... does not.
Now the change in DtaExecutionHost:
--- before\Microsoft.TeamFoundation.TestClient.PublishTestResults-19.204.32418.1\Microsoft.TeamFoundation.TestClient.PublishTestResults\TrxResultParser.cs Mon Oct 03 16:56:16 2022
+++ after\Microsoft.TeamFoundation.TestClient.PublishTestResults-19.210.32906.4\Microsoft.TeamFoundation.TestClient.PublishTestResults\TrxResultParser.cs Mon Oct 03 16:55:38 2022
@@ -229,9 +229,9 @@
}
XmlNode testResultNode = definitionNode.SelectSingleNode("./TestMethod");
string automatedTestName = null;
- if (testResultNode?.Attributes["className"] != null && testResultNode.Attributes["name"] != null)
+ if (testResultNode?.Attributes["className"] != null && (definitionNode.Attributes["name"]?.Value != null || testResultNode.Attributes["name"].Value != null))
{
- string testCaseName = testResultNode.Attributes["name"].Value;
+ string testCaseName = ((definitionNode.Attributes["name"]?.Value == null) ? testResultNode.Attributes["name"].Value : definitionNode.Attributes["name"].Value);
string className = testResultNode.Attributes["className"].Value;
automatedTestName = ((testCaseName.StartsWith(className) && testCaseName.Contains(".")) ? testCaseName : string.Join(".", className, testCaseName));
if (_chutzpahTestFileExtensions.Contains(storageExtension?.Trim(), StringComparer.OrdinalIgnoreCase) && !string.IsNullOrEmpty(storageFileName))This starts overriding the test case name from the name attribute on the <UnitTest/>.
Before: automatedTestName = TestMethod["className'"] + "." + TestMethod["name"]
After: automatedTestName = TestMethod["className"] + "." + (UnitTest["name"] ?? TestMethod["name"])
In practice, instead of making the rerun as:
VSTest.Console.exe /TestCaseFilter:"FullyQualifiedName=Example.Project.Tests.TestClass.TestMethod"
and unfortunately rerunning all variants of the test, but at least working, it now results in:
VSTest.Console.exe /TestCaseFilter:"FullyQualifiedName=Example.Project.Tests.TestClass.TestMethod (One)"
Which does not work at all.
It is wrong for two reasons:
- The paretheses are not escaped correctly
Incorrect format for TestCaseFilter Missing Operator '|' or '&'. Specify the correct format and try again. Note that the incorrect format can lead to no test getting executed. - It is not a valid FullyQualifiedName, that needs really just the method FQN. Comparing the display name or whatever would need a completely different filter being made.
For Ben above, it is even simpler - just a test display name with spaces, resulting in
FullyQualifiedName=JourneyTests.Features.HappyPathFeature.New user sign up journey
instead of a proper method name. Again resulting in no test being found to retry.
bendursley
Mortana89
LURKEN
MBGoldberg-BD
fravaee
2DORSC
swapmali99
Morten4
DavidDanaj
vinayakmsft
nathan-j-nd
ismael-ms
eyalLefler
dogirala
Liran-Dobrish
sablanc-msft
jaydeboer
SeMuell
We've noticed in the last few days that a couple of our VS Test runs no longer retries on failures, and upon retry, it is unable to find the name spaces for those tests.
The tests are written in SpecFlow 3.9.74. We were using an old Test SDK (16.5) but on upgrade to 17.3.2, we still have this issue. We are running on Windows-Latest (2022). Tried with 2019 but had the same issue as well. We did not use VS Installer first, but I've tried with that, and we still have the issue where the retries no longer work
We run using the below YAML code:
- task: VSTest@2 displayName: 'VsTest - testAssemblies' inputs: testAssemblyVer2: | **\JourneyTests.dll !**\*TestAdapter.dll !**\obj\** rerunFailedTests: true rerunFailedThreshold: 35 rerunMaxAttempts: 2 testRunTitle: '${{ parameters.Environment }} - Journey Tests' searchFolder: $(PIPELINE.WORKSPACE)\${{ parameters.PipelineResource }}\${{ parameters.ArtifactResource }} ${{ if ne(parameters.CategoryFilter, '') }}: testFiltercriteria: ${{ parameters.CategoryFilter }} env: ${{ if eq(parameters.UseMocks, true) }}: UseMocks: true ${{ else }}: UseMocks: falseIn our logs, you can see it start the retry attempt, then fail as it is unable to match the name space
No test matches the given testcase filter
FullyQualifiedName=JourneyTests.Features.HappyPathFeature.New user sign up journey.This seems to have changed sometime between Tuesday of this week and Wednesday of this week. We run these tests on a schedule, so there was no code change between the runs on Tuesday and Wednesday.
Tue:
Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220626-01 (x64)[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET Core 3.1.29)DTA Execution Host 19.210.32906.4Wed:
Microsoft (R) Test Execution Command Line Tool Version 17.3.0-preview-20220626-01 (x64)[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET Core 3.1.29)DTA Execution Host 19.204.32418.1Environment