search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.51k stars 2.62k forks source link

[REGRESSION]: VSTest: Change from TestCaseFilter:FullyQualifiedName to TestCaseFilter:ClassName&Name breaks Rerun Failed Tests option #20539

Open aaron-luptak opened 1 month ago

aaron-luptak commented 1 month ago

New issue checklist

Task name

VSTest

Breaking task version

3.246.1

Last working task version

3.231.0

Regression Description

Getting

System.Exception: No tests were discovered while attempting to rerun failed tests. This is likely to happen if these tests are xunit or nunit datadriven/parameterized tests or tests with custom display names with spaces or special characters which are not supported with rerun.

on a pipeline where test rerun was previously working.

Tests are MSTest, not datadriven or parameterized, no custom display names.

Classic Pipeline, Windows 7 agent.

Seems to be a change in the TestCaseFilter arugment: /TestCaseFilter:"FullyQualifiedName=TestCxEpjSerializer.CxEpjSerializerTests.GetEvents_MeetsPerformanceExpectations" - works /TestCaseFilter:"(ClassName=TestSECSConnect.CxSMSTests&Name=SECSI_NormalMessaging_Works)" - does not work

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 7

Relevant log output

2024-10-11T20:32:08.3832790Z **** Rerunning failed tests for Test run 1095563 * 2024-10-11T20:32:08.3832790Z Max attempts: 3; Current attempt: 1; Failed test cases threshold: 30; Failed test cases max limit: 0 2024-10-11T20:32:08.3832790Z [command]C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "@C:\Temp\buildagent_work_temp\1k2kqbofe5a.tmp" 2024-10-11T20:32:08.4929018Z Microsoft (R) Test Execution Command Line Tool Version 16.11.0 2024-10-11T20:32:08.4929018Z Copyright (c) Microsoft Corporation. All rights reserved. 2024-10-11T20:32:08.4929018Z vstest.console.exe /TestCaseFilter:"(ClassName=TestSECSConnect.CxSMSTests&Name=SECSI_NormalMessaging_Works)" 2024-10-11T20:32:08.4929018Z "C:\Temp\buildagent_work\4\s\Source\Shared\TestSECSConnect.dll" 2024-10-11T20:32:08.4929018Z /Settings:"C:\Temp\buildagent_work\4\s\Source\dcbmssrluio.tmp.runsettings" 2024-10-11T20:32:08.4929018Z /Logger:"trx" 2024-10-11T20:32:08.4929018Z /TestAdapterPath:"C:\Temp\buildagent_work\4\s" 2024-10-11T20:32:08.4929018Z /Diag:"C:\Temp\buildagent_work_temp\Attempt-3_drsndv.diag" 2024-10-11T20:32:08.4929018Z /logger:trx /platform:x64 2024-10-11T20:32:08.8687514Z Starting test execution, please wait... 2024-10-11T20:32:08.8687514Z Logging Vstest Diagnostics in file: C:\Temp\buildagent_work_temp\Attempt-3_drsndv.diag 2024-10-11T20:32:09.0096950Z A total of 1 test files matched the specified pattern. 2024-10-11T20:32:09.9806398Z The test execution was delegated to legacy TestPlatform runner. You are seeing this message most likely because you are running MSTest v1 tests with a testsettings file, Generic tests, Ordered tests or CodedUI tests. All of those test types, and the legacy TestPlatform runner are legacy. It is recommended that you migrate to MSTest v2. 2024-10-11T20:32:10.2312062Z TestSettings are legacy. It is recommended that you use RunSettings to configure test runs. To learn more, see http://aka.ms/runsettings 2024-10-11T20:32:10.5600746Z No test matches the given testcase filter (ClassName=TestSECSConnect.CxSMSTests&Name=SECSI_NormalMessaging_Works) in C:\Temp\buildagent_work\4\s\Source\Shared\TestSECSConnect.dll 2024-10-11T20:32:10.6853578Z Results File: C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx 2024-10-11T20:32:12.9874366Z ##[debug]Exited vstest.console.exe with code 0. 2024-10-11T20:32:12.9874366Z ##[debug]PERF: ExecuteVsTestPhase.InvokeVSTest: took 4577.1487 ms 2024-10-11T20:32:12.9874366Z Vstest.console.exe exited with code 0. 2024-10-11T20:32:12.9874366Z **** Completed test execution * 2024-10-11T20:32:12.9874366Z ##[debug]Current Phase: MS.VS.TestService.VstestConsoleAdapter.ExecuteVsTestPhase Phase Result: MS.VS.TestService.VstestConsoleAdapter.ExecutionVsTestPhaseResults 2024-10-11T20:32:12.9874366Z ##[debug]PERF WARNING: Running the phase MS.VS.TestService.VstestConsoleAdapter.ExecuteVsTestPhase: took 4577.2488 ms 2024-10-11T20:32:12.9874366Z ##[debug]Current phase: MS.VS.TestService.VstestConsoleAdapter.PublishTestResultPhase 2024-10-11T20:32:12.9874366Z ##[debug]Starting Publish Test Results: TIA: False 2024-10-11T20:32:12.9874366Z ##[debug]Starting PublishTestResult from Result Files 2024-10-11T20:32:12.9874366Z Test results files: C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx 2024-10-11T20:32:12.9874366Z ##[debug]Updating Test results: 1095563 2024-10-11T20:32:12.9874366Z ##[debug]Entering ParseTestResultFiles 2024-10-11T20:32:12.9874366Z ##[debug]runContext.ReleaseURI is null 2024-10-11T20:32:12.9874366Z ##[debug]runContext.ReleaseEnvironmentUri is null 2024-10-11T20:32:12.9874366Z ##[debug]Reading test results from file 'C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx'. 2024-10-11T20:32:12.9874366Z ##[debug]Setting run start and finish times. 2024-10-11T20:32:12.9874366Z ##[debug]Attachment location: C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10\In. 2024-10-11T20:32:12.9874366Z No Result Found to Publish 'C:\Temp\buildagent_work\4\s\TestResults\QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx'. 2024-10-11T20:32:12.9874366Z ##[debug]Total test results: 0. 2024-10-11T20:32:12.9874366Z ##[debug]Leaving ParseTestResultFiles 2024-10-11T20:32:12.9874366Z Updating test results: 0 2024-10-11T20:32:12.9874366Z ##[debug]Entering UpdateTestResultsAsync 2024-10-11T20:32:13.0500782Z Publishing test results to test run '1095563'. 2024-10-11T20:32:13.6139822Z ##[debug]PERF: GetTestCaseResults: took 544.585 ms 2024-10-11T20:32:13.6139822Z ##[debug]PERF WARNING: GetTestCaseResults: took 544.585 ms 2024-10-11T20:32:13.6139822Z TestResults To Publish 0, Test run id:1095563 2024-10-11T20:32:13.6139822Z ##[debug]Leaving UpdateTestResultsAsync 2024-10-11T20:32:13.6139822Z Updated test results: 0 2024-10-11T20:32:13.6139822Z Publishing Attachments: 1 2024-10-11T20:32:13.6139822Z ##[debug]Entering CreateTestRunAttachmentsAsync 2024-10-11T20:32:13.6766430Z ##[debug]Leaving CreateTestRunAttachmentsAsync 2024-10-11T20:32:13.6766430Z ##[debug]Uploading test run attachements individually 2024-10-11T20:32:13.6766430Z ##[debug]Entering UploadTestRunLogAsync 2024-10-11T20:32:13.6766430Z ##[debug]Blob name: GeneralAttachment/QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx . 2024-10-11T20:32:13.8646254Z ##[debug]File uploaded successsfully on LogStore QEBuildService_QEBACOMMON_2024-10-11_14_32_10.trx 2024-10-11T20:32:13.8646254Z ##[debug]Completed PublishTestResult 2024-10-11T20:32:13.8646254Z ##[debug]PERF: PublishTestResultPhase.Publish: took 890.0333 ms 2024-10-11T20:32:13.8646254Z ##[debug]Completed Publish Test Results. 2024-10-11T20:32:13.8646254Z ##[debug]Current Phase: MS.VS.TestService.VstestConsoleAdapter.PublishTestResultPhase Phase Result: MS.VS.TestService.VstestConsoleAdapter.PublishTestResultPhaseResults 2024-10-11T20:32:13.8646254Z ##[error]Error occurred while publishing test results : System.Exception: No tests were discovered while attempting to rerun failed tests. This is likely to happen if these tests are xunit or nunit datadriven/parameterized tests or tests with custom display names with spaces or special characters which are not supported with rerun. Please reach out to us at https://github.com/microsoft/azure-pipelines-tasks/issues if you are hitting this for tests other than the ones mentioned before. 2024-10-11T20:32:13.8646254Z ##[debug]Processed: ##vso[task.logissue type=error;]Error occurred while publishing test results : System.Exception: No tests were discovered while attempting to rerun failed tests. This is likely to happen if these tests are xunit or nunit datadriven/parameterized tests or tests with custom display names with spaces or special characters which are not supported with rerun. Please reach out to us at https://github.com/microsoft/azure-pipelines-tasks/issues if you are hitting this for tests other than the ones mentioned before.

Full task logs with system.debug enabled

unsuccess_log.txt success_log.txt

Repro steps

No response

aaron-luptak commented 1 month ago

A few things we've tried that have not resolved the issue:

I have been able to work around the issue by changing the "Batch tests" option under "Advanced execution options" to "Based on test assemblies" - in this case, it is able to rerun failed tests.