search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.45k stars 2.6k forks source link

VSTestV2 task success when failed to run any test due to wrong calculation in flaky logic #17863

Open puagarwa opened 1 year ago

puagarwa commented 1 year ago

Question, Bug, or Feature? Bug Type:

Enter Task Name: VSTESTV2

Environment

Issue Description

VSTESTV2 task success when failed to run any test due to some base failure like .net framework missing in agent. Since it find there are no test, it end up marking the task as success as it find number of failed test as zero I tried enabling option "Fail the task if minimum number of tests are not run" with 1 to make sure task fail when no test runs but it actually think its case of all failure test being flaky and hence marked the task as success.

Task logs

2023-03-03T10:19:38.6409022Z ##[error]Test Run Aborted. 2023-03-03T10:19:38.6746599Z Vstest.console.exe exited with code 1. 2023-03-03T10:19:38.6747155Z **** Completed test execution ***** 2023-03-03T10:19:38.6827763Z Test results files: D:\a_temp\TestResults\VssAdministrator_WIN-0JI1D5IMURB_2023-03-03_10_19_38.trx 2023-03-03T10:19:38.7204402Z No Result Found to Publish 'D:\a_temp\TestResults\VssAdministrator_WIN-0JI1D5IMURB_2023-03-03_10_19_38.trx'. 2023-03-03T10:19:39.1679564Z Created test run: 69819915 2023-03-03T10:19:39.1685027Z Publishing test results: 0 2023-03-03T10:19:39.2763684Z Publishing test results to test run '69819915'. 2023-03-03T10:19:39.2799770Z TestResults To Publish 0, Test run id:69819915 2023-03-03T10:19:39.2800138Z Published test results: 0 2023-03-03T10:19:39.2810315Z Publishing Attachments: 1 2023-03-03T10:19:39.8920046Z ##[error]The specified minimum number of tests 1 were not executed in the test run. 2023-03-03T10:19:39.9863778Z Execution Result Code 1 is non zero, checking for failed results 2023-03-03T10:19:43.2191141Z Failed tests if any were marked as flaky hence changing the exit code to 0 2023-03-03T10:19:43.2191809Z PERF WARNING: TestRunCompletionPhase:GetTestRunOutcomeFromPublishTestResultLibrary: took 3171.0082 ms 2023-03-03T10:19:43.2480067Z Completed TestExecution Model... 2023-03-03T10:19:43.2825307Z ##[section]Finishing: Upload Service Integration Test

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

2023-03-03T10:19:39.8920046Z ##[error]The specified minimum number of tests 1 were not executed in the test run. 2023-03-03T10:19:39.9863778Z Execution Result Code 1 is non zero, checking for failed results 2023-03-03T10:19:43.2191141Z Failed tests if any were marked as flaky hence changing the exit code to 0

puagarwa commented 1 year ago

And just to mention, i know there is an option to turn off flaky feature at project setting level, but i cant do that because that would men turning flaky feature off for entire division. so thats not an option. beside this is clearly a bug.

vinayakmsft commented 1 year ago

@puagarwa Thanks for identifying this. We will log this as bug in our sprint work items and take it up in the next sprint. Also, can you please share any pipeline where you have faced this issue? This would help in reproducing the scenario and investigate further.

puagarwa commented 1 year ago

shared privately.

puagarwa commented 1 year ago

HI @vinayakmsft any update on this ?

meghandaly commented 1 year ago

@vinayakmsft My team is also encountering this issue and we nearly checked in a regression as a result. False confidence in faulty pipelines is a dangerous thing.

image

Can you please update @puagarwa and I on the current status?

dogirala commented 11 months ago

Created an internal workitem for this, will prioritize in sprint 229

mashtabiei-fourthline commented 10 months ago

Thanks! Is there any workaround available in the meantime?

github-actions[bot] commented 2 months ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

puagarwa commented 2 months ago

@dogirala Any update on the issue.

dogirala commented 2 months ago

This has been fixed with PR --> https://dev.azure.com/mseng/AzureDevOps/_git/AzureDevOps/pullrequest/754887, FeatureFlag has to be turned on. @chandrashritii is currently rolling it out ring wise and monitoring for any errors.