search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.49k stars 2.61k forks source link

[BUG]: Automated tests from test plans fail at VSTest task not discovering tests when executed #19237

Closed sailesh89 closed 5 months ago

sailesh89 commented 11 months ago

Task name

VsTest

Task version

2.x

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows

Task log

2023-11-06T20:25:00.4717439Z ==============================================================================
2023-11-06T20:25:00.4717748Z Task         : Visual Studio Test
2023-11-06T20:25:00.4717891Z Description  : Run unit and functional tests (Selenium, Appium, Coded UI test, etc.) using the Visual Studio Test (VsTest) runner. Test frameworks that have a Visual Studio test adapter such as MsTest, xUnit, NUnit, Chutzpah (for JavaScript tests using QUnit, Mocha and Jasmine), etc. can be run. Tests can be distributed on multiple agents using this task (version 2).
2023-11-06T20:25:00.4719605Z Version      : 2.229.0
2023-11-06T20:25:00.4719715Z Author       : Microsoft Corporation
2023-11-06T20:25:00.4719870Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/test/vstest
2023-11-06T20:25:00.4720201Z ==============================================================================
2023-11-06T20:25:01.4045017Z In distributed testing flow
2023-11-06T20:25:01.4078805Z ======================================================
2023-11-06T20:25:01.4099576Z Test selector : Test run
2023-11-06T20:25:01.4101677Z Test run Id : '9999999'
2023-11-06T20:25:01.4103302Z Search folder : C:\a\r1\a
2023-11-06T20:25:01.4273716Z Action when minimum tests threshold not met : donothing
2023-11-06T20:25:01.4275354Z Minimum tests expected to be run: 0
2023-11-06T20:25:01.7381653Z Distributed test execution, number of agents in job : 1
2023-11-06T20:25:01.7394818Z Run settings file : C:\a\r1\a\__artifactAlias\Test.Automation.xxxxxxxxxxxx\test.runsettings
2023-11-06T20:25:01.7400384Z Run in parallel : false
2023-11-06T20:25:01.7402004Z Run in isolation : false
2023-11-06T20:25:01.7403679Z Path to custom adapters : undefined
2023-11-06T20:25:01.7407979Z Other console options : /diag:C:\a\_temp\TestResults\diagnostic.txt
2023-11-06T20:25:01.7462887Z ##[warning]Other console options is not supported for this task configuration. This option will be ignored.
2023-11-06T20:25:01.7475340Z Code coverage enabled : false
2023-11-06T20:25:01.7477895Z Diagnostics enabled : true
2023-11-06T20:25:01.7482579Z Rerun failed tests: true
2023-11-06T20:25:01.7485799Z Rerun failed tests threshold: 30
2023-11-06T20:25:01.7487520Z Rerun maximum attempts: 3
2023-11-06T20:25:01.7612050Z ======================================================
2023-11-06T20:25:01.7616615Z Source filter: **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**
2023-11-06T20:25:02.8406336Z ##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.
2023-11-06T20:25:02.8558104Z ##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.
2023-11-06T20:25:03.0655309Z ##[warning]Test Impact (Run only Impacted tests) and Code Coverage data collector will not work.
2023-11-06T20:25:03.0762832Z [command]C:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\2.229.0\Modules\DTAExecutionHost.exe --inputFile C:\a\_temp\input_90e405e0-7ce2-11ee-a7d4-5fbeb6c44523.json
2023-11-06T20:25:03.1460510Z ##########################################################################
2023-11-06T20:25:03.1461947Z DtaExecutionHost version 19.229.34210.1.
2023-11-06T20:25:04.8620369Z ===========================================
2023-11-06T20:25:04.8621522Z AgentName: agenttest-agenttest-63040
2023-11-06T20:25:04.8622439Z ServiceUrl: https://dev.azure.com/ORG/
2023-11-06T20:25:04.8623207Z TestPlatformVersion: 17.7.2
2023-11-06T20:25:04.8624085Z EnvironmentUri: vstest://env/PROJECT/_apis/release/777/88888/000000/1
2023-11-06T20:25:04.8625204Z QueryForTaskIntervalInMilliseconds: 3000
2023-11-06T20:25:04.8626051Z MaxQueryForTaskIntervalInMilliseconds: 10000
2023-11-06T20:25:04.8626687Z QueueNotFoundDelayTimeInMilliseconds: 3000
2023-11-06T20:25:04.8627426Z MaxQueueNotFoundDelayTimeInMilliseconds: 50000
2023-11-06T20:25:04.8628353Z ===========================================
2023-11-06T20:25:05.4883421Z Result Attachments will be stored in LogStore
2023-11-06T20:25:05.5030430Z Run Attachments will be stored in LogStore
2023-11-06T20:25:05.6587930Z TestExecutionHost.Execute: Registered TestAgent : 54119 : agenttest-agenttest-63040
2023-11-06T20:25:05.6804108Z Provided settings file:
2023-11-06T20:25:05.6821804Z <?xml version="1.0" encoding="utf-8"?>
2023-11-06T20:25:05.6822728Z <RunSettings>
2023-11-06T20:25:05.6823430Z    <!-- Configurations that affect the Test Framework -->
2023-11-06T20:25:05.6824409Z    <RunConfiguration>
2023-11-06T20:25:05.6825600Z        <DisableParallelization>false</DisableParallelization>
2023-11-06T20:25:05.6826515Z        <MaxCpuCount>1</MaxCpuCount>
2023-11-06T20:25:05.6827182Z    </RunConfiguration>
2023-11-06T20:25:05.6827705Z    <DataCollectionRunSettings>
2023-11-06T20:25:05.6829232Z        <DataCollectors>
2023-11-06T20:25:05.6833679Z            <DataCollector uri="datacollector://microsoft/VideoRecorder/1.0" assemblyQualifiedName="Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder.VideoRecorderDataCollector, Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" friendlyName="Screen and Voice Recorder">
2023-11-06T20:25:05.6838490Z                <!--Video data collector was introduced in Visual Studio 2017 version 15.5--> 
2023-11-06T20:25:05.6839988Z                <Configuration>
2023-11-06T20:25:05.6841487Z                     <!--Set "sendRecordedMediaForPassedTestCase" to "false" to add video attachments to failed tests only--> 
2023-11-06T20:25:05.6843404Z                    <MediaRecorder sendRecordedMediaForPassedTestCase="false"  xmlns="">
2023-11-06T20:25:05.6853458Z                        ​
2023-11-06T20:25:05.6854664Z                        <ScreenCaptureVideo bitRate="512" frameRate="2" quality="20" />​
2023-11-06T20:25:05.6855808Z                    </MediaRecorder>​
2023-11-06T20:25:05.6856397Z                </Configuration>
2023-11-06T20:25:05.6856925Z            </DataCollector>
2023-11-06T20:25:05.6857466Z        </DataCollectors>
2023-11-06T20:25:05.6857991Z    </DataCollectionRunSettings>
2023-11-06T20:25:05.6858446Z 
2023-11-06T20:25:05.6859068Z    <!-- MSTest adapter -->
2023-11-06T20:25:05.6859650Z    <MSTest>
2023-11-06T20:25:05.6860082Z        <Parallelize>
2023-11-06T20:25:05.6860603Z            <Workers>1</Workers>
2023-11-06T20:25:05.6861361Z            <Scope>MethodLevel</Scope>
2023-11-06T20:25:05.6861992Z        </Parallelize>
2023-11-06T20:25:05.6862519Z    </MSTest>
2023-11-06T20:25:05.6862981Z </RunSettings>
2023-11-06T20:25:05.6863291Z 
2023-11-06T20:25:05.6975334Z IsValidServiceResponse: Received None command..Service Workflow is not active
2023-11-06T20:25:05.7126093Z Updated Run Settings:
2023-11-06T20:25:05.7126936Z <RunSettings>
2023-11-06T20:25:05.7128560Z   <!-- Configurations that affect the Test Framework -->
2023-11-06T20:25:05.7129565Z   <RunConfiguration>
2023-11-06T20:25:05.7130385Z     <DisableParallelization>false</DisableParallelization>
2023-11-06T20:25:05.7132110Z     <MaxCpuCount>1</MaxCpuCount>
2023-11-06T20:25:05.7132889Z     <BatchSize>1000</BatchSize>
2023-11-06T20:25:05.7133939Z     <ResultsDirectory>C:\a\_temp\TestResults</ResultsDirectory>
2023-11-06T20:25:05.7135539Z   </RunConfiguration>
2023-11-06T20:25:05.7136593Z   <DataCollectionRunSettings>
2023-11-06T20:25:05.7137390Z     <DataCollectors>
2023-11-06T20:25:05.7141811Z       <DataCollector uri="datacollector://microsoft/VideoRecorder/1.0" assemblyQualifiedName="Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder.VideoRecorderDataCollector, Microsoft.VisualStudio.TestTools.DataCollection.VideoRecorder, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" friendlyName="Screen and Voice Recorder">
2023-11-06T20:25:05.7148324Z         <!--Video data collector was introduced in Visual Studio 2017 version 15.5-->
2023-11-06T20:25:05.7149933Z         <Configuration>
2023-11-06T20:25:05.7151568Z           <!--Set "sendRecordedMediaForPassedTestCase" to "false" to add video attachments to failed tests only-->
2023-11-06T20:25:05.7153701Z           <MediaRecorder sendRecordedMediaForPassedTestCase="false" xmlns="">
2023-11-06T20:25:05.7155995Z                        ​
2023-11-06T20:25:05.7157022Z                        <ScreenCaptureVideo bitRate="512" frameRate="2" quality="20" />​
2023-11-06T20:25:05.7158314Z                    </MediaRecorder>​
2023-11-06T20:25:05.7159299Z                </Configuration>
2023-11-06T20:25:05.7160126Z       </DataCollector>
2023-11-06T20:25:05.7161238Z       <DataCollector friendlyName="blame" enabled="True">
2023-11-06T20:25:05.7162481Z         <Configuration>
2023-11-06T20:25:05.7163601Z           <ResultsDirectory>C:\a\_temp\TestResults</ResultsDirectory>
2023-11-06T20:25:05.7164956Z           <CollectDump CollectAlways="true" />
2023-11-06T20:25:05.7166559Z           <CollectDump />
2023-11-06T20:25:05.7167971Z         </Configuration>
2023-11-06T20:25:05.7169062Z       </DataCollector>
2023-11-06T20:25:05.7170326Z     </DataCollectors>
2023-11-06T20:25:05.7171718Z   </DataCollectionRunSettings>
2023-11-06T20:25:05.7172864Z   <!-- MSTest adapter -->
2023-11-06T20:25:05.7173735Z   <MSTest>
2023-11-06T20:25:05.7174906Z     <Parallelize>
2023-11-06T20:25:05.7175851Z       <Workers>1</Workers>
2023-11-06T20:25:05.7177126Z       <Scope>MethodLevel</Scope>
2023-11-06T20:25:05.7178766Z     </Parallelize>
2023-11-06T20:25:05.7179778Z   </MSTest>
2023-11-06T20:25:05.7180651Z   <LoggerRunSettings>
2023-11-06T20:25:05.7181586Z     <Loggers>
2023-11-06T20:25:05.7182653Z       <Logger friendlyName="blame" enabled="True" />
2023-11-06T20:25:05.7184883Z     </Loggers>
2023-11-06T20:25:05.7186072Z   </LoggerRunSettings>
2023-11-06T20:25:05.7186992Z </RunSettings>
2023-11-06T20:25:05.8136769Z Creating run for selected test assemblies with following parameters
2023-11-06T20:25:05.8139134Z SourceFilter: *.dll TestCaseFilter: 
2023-11-06T20:25:05.8141563Z Run title: xxxxxxxxxxxxxxxxxxxxxxx - Assembly_Release-13
2023-11-06T20:25:05.8142865Z Build location: C:\a\r1\a
2023-11-06T20:25:05.8143862Z Build Id: 3333333
2023-11-06T20:25:06.0355469Z Test run with Id 9999999 associated
2023-11-06T20:25:16.7384975Z Received the command : Start
2023-11-06T20:25:16.7404361Z TestExecutionHost.ProcessCommand. Start Command handled
2023-11-06T20:25:18.6282525Z Slice with id = 66666, of type = 'Execution' received.
2023-11-06T20:25:18.8977548Z Count of test sources found: 2
2023-11-06T20:25:18.9210177Z =================================================================
2023-11-06T20:25:18.9211674Z Discovering tests from sources
2023-11-06T20:25:18.9461578Z Using new test platform for test execution
2023-11-06T20:25:18.9571175Z Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
2023-11-06T20:25:29.3985379Z DiscoveryMessage : [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.3.1+6b60a9e56a (64-bit .NET 6.0.16)
2023-11-06T20:25:30.4142575Z DiscoveryMessage : [xUnit.net 00:00:00.56]   Discovering: xxxxxxxxxxxx.FunctionalTests
2023-11-06T20:25:30.8324559Z DiscoveryMessage : [xUnit.net 00:00:01.45]   Discovered:  xxxxxxxxxxxx.FunctionalTests
2023-11-06T20:25:33.1196296Z Number of testcases discovered : 707
2023-11-06T20:25:33.1395128Z Discovered tests 707 from sources
2023-11-06T20:25:33.1396036Z =================================================================
2023-11-06T20:25:33.1641445Z [RunStatistics]This execution slice with id '66666', received '1' testcases to execute out of which '0' is discovered.
2023-11-06T20:25:33.1967140Z List of test cases in slice with id: 66666
2023-11-06T20:25:33.1971615Z Source: xxxxxxxxxxxx.FunctionalTests.dll, FullyQualifiedTestName: xxxxxxxxxxxx.FunctionalTests.Features.Functional.xxxxxxxxxxxx.SAMPLETESTMETHOD
2023-11-06T20:25:33.2229981Z ##[error]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2023-11-06T20:25:33.2237947Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:33.2241541Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:33.2290505Z ##[error]The slice of type 'Execution' is 'Aborted' because of the error : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2023-11-06T20:25:33.2309860Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:33.2313698Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:37.7462112Z Received the command : Stop
2023-11-06T20:25:37.7463428Z TestExecutionHost.ProcessCommand. Stop Command handled
2023-11-06T20:25:37.7472402Z SliceFetch Aborted. Moving to the TestHostEnd phase
2023-11-06T20:25:37.7936092Z Please use this link to analyze the test run : https://dev.azure.com/ORG/PROJECT/_TestManagement/Runs#_a=resultQuery&runId=9999999&queryPath=Recent+Run%2FRun9999999
2023-11-06T20:25:37.7939786Z Test run '9999999' is in 'Aborted' state with 'Total Tests' : 1 and 'Passed Tests' : 0.
2023-11-06T20:25:37.7996602Z ##[error]Test run is aborted. Logging details of the run logs.
2023-11-06T20:25:37.8005110Z ##[error]System.Exception: The test run was aborted, failing the task.
2023-11-06T20:25:37.9039187Z ##########################################################################
2023-11-06T20:25:37.9603912Z ##[section]Finishing: VsTest - testRun

Relevant log output

2023-11-06T20:25:06.0355469Z Test run with Id 9999999 associated
2023-11-06T20:25:16.7384975Z Received the command : Start
2023-11-06T20:25:16.7404361Z TestExecutionHost.ProcessCommand. Start Command handled
2023-11-06T20:25:18.6282525Z Slice with id = 66666, of type = 'Execution' received.
2023-11-06T20:25:18.8977548Z Count of test sources found: 2
2023-11-06T20:25:18.9210177Z =================================================================
2023-11-06T20:25:18.9211674Z Discovering tests from sources
2023-11-06T20:25:18.9461578Z Using new test platform for test execution
2023-11-06T20:25:18.9571175Z Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created
2023-11-06T20:25:29.3985379Z DiscoveryMessage : [xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.5.3.1+6b60a9e56a (64-bit .NET 6.0.16)
2023-11-06T20:25:30.4142575Z DiscoveryMessage : [xUnit.net 00:00:00.56]   Discovering: xxxxxxxxxxxx.FunctionalTests
2023-11-06T20:25:30.8324559Z DiscoveryMessage : [xUnit.net 00:00:01.45]   Discovered:  xxxxxxxxxxxx.FunctionalTests
2023-11-06T20:25:33.1196296Z Number of testcases discovered : 707
2023-11-06T20:25:33.1395128Z Discovered tests 707 from sources
2023-11-06T20:25:33.1396036Z =================================================================
2023-11-06T20:25:33.1641445Z [RunStatistics]This execution slice with id '66666', received '1' testcases to execute out of which '0' is discovered.
2023-11-06T20:25:33.1967140Z List of test cases in slice with id: 66666
2023-11-06T20:25:33.1971615Z Source: xxxxxxxxxxxx.FunctionalTests.dll, FullyQualifiedTestName: xxxxxxxxxxxx.FunctionalTests.Features.Functional.xxxxxxxxxxxx.SAMPLETESTMETHOD
2023-11-06T20:25:33.2229981Z ##[error]ExecutionAndPublish.Run : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2023-11-06T20:25:33.2237947Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:33.2241541Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:33.2290505Z ##[error]The slice of type 'Execution' is 'Aborted' because of the error : Microsoft.VisualStudio.TestService.VstestAdapter.TestsNotFoundException: No test assemblies found on the test machine matching the source filter criteria or no tests discovered matching test filter criteria. Verify that test assemblies are present on the machine and test filter criteria is correct.
2023-11-06T20:25:33.2309860Z    at Microsoft.VisualStudio.TestService.VstestAdapter.Execution.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:33.2313698Z    at Microsoft.VisualStudio.TestService.VstestAdapter.ExecutionAndPublish.Run(ExecutionStateContext stateModelContext, CancellationToken cancellationToken)
2023-11-06T20:25:37.7462112Z Received the command : Stop
2023-11-06T20:25:37.7463428Z TestExecutionHost.ProcessCommand. Stop Command handled

Aditional info

Looks like it started to happen after the VSTest task is updated to 2.229.0. I ca see the task was running fine till when it was 2.228.0. after that the runs were failing.
mikeblakeuk commented 11 months ago

Related to https://github.com/microsoft/azure-pipelines-tasks/issues/19222. We have see it work and fail with with 2.229.0.

Working

Discovered tests 12348 from sources
=================================================================
[RunStatistics]This execution slice with id '168030', received '500' testcases to execute out of which '500' is discovered.
**************** Starting test execution *********************
Using new test platform for test execution
Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper instance will be created

Broken

Discovered tests 12371 from sources
=================================================================
[RunStatistics]This execution slice with id '168529', received '500' testcases to execute out of which '0' is discovered.
List of test cases in slice with id: 168529
mikeblakeuk commented 11 months ago

Possible fix is to move from VSTest@2 to VSTest@3 (Undocumented https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines)

Wonder if the node pathing has changed as it looks for the dlls? Could be due to https://github.com/microsoft/azure-pipelines-tasks/pull/19215

dlaw1912 commented 11 months ago

Possible fix is to move from VSTest@2 to VSTest@3 (Undocumented https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines)

I have a similar issue where 2.229.0 was working (found and executed 6/6 test cases in a Test Plan), but now only runs 3/6 Test Cases and leaves the remaining test cases with an Outcome of 'In Progress'.

I have just tried VSTest@3 (v3.225.0), and now all 6 test cases are executed successfully again.

sailesh89 commented 11 months ago

Possible fix is to move from VSTest@2 to VSTest@3 (Undocumented https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/vstest-v2?view=azure-pipelines)

Wonder if the node pathing has changed as it looks for the dlls? Could be due to #19215

Upgrade won't work in our case as it will fail the Automated test run from the Test plan validation. Looks like its when we select a subset of tests to be run from test plan, choosing this release with vstest@v2 only works. Rest fails the validation and wont kick it off. Link to referenced steps from microsoft. AutomatedTestRun

sailesh89 commented 11 months ago

I did test with vstest@V3 to run entire test plan. It seems to be working with the v3 task. But unfortunately, the Automated test run validation is not allowing V3 task. Probably will need to fix that validation and we can try with the V3 task in the test run mode from Automated test run.

mikeblakeuk commented 11 months ago

@qianz2 @merlynomsft @rasunkar any thoughts?

DuryDuty commented 11 months ago

I have exactly the same setup as @sailesh89 and am experiencing the same issue.

On the afternoon of 9th November a number of tests stopped working. Now this morning (10th) those same tests are executing. No changes were made on my end at any point.

github-actions[bot] commented 5 months ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

ofthedoor commented 4 months ago

This still seems to be an issue with Version 2 of the vstest runner,

still unable to run tests from test plan with version 3

can we reopen this issue?

or have some form of update on this issue?

mikeblakeuk commented 4 months ago

I stopped using VSTest it as it doesn't work on Ubuntu anyway. We use dotnet test now and our own custom version of https://github.com/idubnori/ParallelTestingSample-dotnet-core