microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.43k stars 2.59k forks source link

[BUG]: DotNetCoreCLI@2 hangs on "dotnet tool update" commands indefinitely with "The STDIO streams did not close within 10 seconds of the exit event from process" #19618

Open Lawls91 opened 4 months ago

Lawls91 commented 4 months ago

New issue checklist

Task name

DotNetCoreCLI

Task version

2

Issue Description

Recently we have had a large uptick in builds getting hung on this step forever until the pipeline times out. I saw related issues that referenced this fix:

https://github.com/microsoft/azure-pipelines-agent/pull/4074

however, it looks like that setting is enabled (although I have not enabled it myself in capabilities, so not sure what exactly has enabled that as the code still shows it default false?). Is there anything we should be doing here to resolve or debug this?

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2022

Relevant log output

2024-03-06T22:08:41.2836461Z ##[section]Starting: Install AzureSignTool
2024-03-06T22:08:41.2977384Z ==============================================================================
2024-03-06T22:08:41.2977843Z Task         : .NET Core
2024-03-06T22:08:41.2977989Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2024-03-06T22:08:41.2978169Z Version      : 2.235.0
2024-03-06T22:08:41.2978329Z Author       : Microsoft Corporation
2024-03-06T22:08:41.2978480Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2024-03-06T22:08:41.2978688Z ==============================================================================
2024-03-06T22:08:45.9535075Z [command]C:\Windows\system32\chcp.com 65001
2024-03-06T22:08:45.9680803Z Active code page: 65001
2024-03-06T22:08:45.9796857Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2024-03-06T22:08:45.9820380Z [command]C:\agent\_work\_tool\dotnet\dotnet.exe tool update --global azuresigntool --version 4.0.1
2024-03-06T22:08:54.4898016Z Tool 'azuresigntool' was reinstalled with the latest stable version (version '4.0.1').
2024-03-06T22:09:04.5270594Z The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent\_work\_tool\dotnet\dotnet.exe'. This may indicate a child process inherited the STDIO streams and has not yet exited.
2024-03-06T22:09:04.5283011Z Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2024-03-07T10:01:29.4560024Z ##[error]The Operation will be canceled. The next steps may not contain expected logs.
2024-03-07T10:01:29.5065729Z ##[error]The operation was canceled.
2024-03-07T10:01:29.5071108Z ##[section]Finishing: Install AzureSignTool

Full task logs with system.debug enabled

Starting: Initialize job
Agent name: 'WIN-XYZ'
Agent machine name: 'WIN-XYZ'
Current agent version: '3.236.0'
Agent running as: 'account'
Prepare build directory.
Set build variables.
Download all required tasks.
Checking job knob settings.
   Knob: DockerActionRetries = true Source: $(VSTSAGENT_DOCKER_ACTION_RETRIES) 
   Knob: AgentEnablePipelineArtifactLargeChunkSize = true Source: $(AGENT_ENABLE_PIPELINEARTIFACT_LARGE_CHUNK_SIZE) 
   Knob: ContinueAfterCancelProcessTreeKillAttempt = true Source: $(VSTSAGENT_CONTINUE_AFTER_CANCEL_PROCESSTREEKILL_ATTEMPT) 
   Knob: ProcessHandlerTelemetry = true Source: $(AZP_75787_ENABLE_COLLECT) 
   Knob: ProcessHandlerEnableNewLogic = true Source: $(AZP_75787_ENABLE_NEW_PH_LOGIC) 
   Knob: IgnoreVSTSTaskLib = true Source: $(AZP_AGENT_IGNORE_VSTSTASKLIB) 
   Knob: FailJobWhenAgentDies = true Source: $(FAIL_JOB_WHEN_AGENT_DIES) 
   Knob: CheckForTaskDeprecation = true Source: $(AZP_AGENT_CHECK_FOR_TASK_DEPRECATION) 
   Knob: MountWorkspace = true Source: $(AZP_AGENT_MOUNT_WORKSPACE) 
Finished checking job knob settings.
Plugin: 'Test Result Parser plugin' is running in background.
Plugin: 'Test File Publisher plugin' is running in background.
Start tracking orphan processes.
Finishing: Initialize job

Starting: Use .NET Core sdk
==============================================================================
Task         : Use .NET Core
Description  : Acquires a specific version of the .NET Core SDK from the internet or the local cache and adds it to the PATH. Use this task to change the version of .NET Core used in subsequent tasks. Additionally provides proxy support.
Version      : 2.235.0
Author       : Microsoft Corporation
Help         : https://aka.ms/AA4xgy0
==============================================================================
Tool to install: .NET Core sdk version 6.0.x.
Found version 6.0.419 in channel 6.0 for user specified version spec: 6.0.x
Version: 6.0.419 was found in cache.
Creating global tool path and pre-pending to PATH.
Finishing: Use .NET Core sdk

2024-03-06T22:08:41.2836461Z ##[section]Starting: Install AzureSignTool
2024-03-06T22:08:41.2977384Z ==============================================================================
2024-03-06T22:08:41.2977843Z Task         : .NET Core
2024-03-06T22:08:41.2977989Z Description  : Build, test, package, or publish a dotnet application, or run a custom dotnet command
2024-03-06T22:08:41.2978169Z Version      : 2.235.0
2024-03-06T22:08:41.2978329Z Author       : Microsoft Corporation
2024-03-06T22:08:41.2978480Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli
2024-03-06T22:08:41.2978688Z ==============================================================================
2024-03-06T22:08:45.9535075Z [command]C:\Windows\system32\chcp.com 65001
2024-03-06T22:08:45.9680803Z Active code page: 65001
2024-03-06T22:08:45.9796857Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version.
2024-03-06T22:08:45.9820380Z [command]C:\agent\_work\_tool\dotnet\dotnet.exe tool update --global azuresigntool --version 4.0.1
2024-03-06T22:08:54.4898016Z Tool 'azuresigntool' was reinstalled with the latest stable version (version '4.0.1').
2024-03-06T22:09:04.5270594Z The STDIO streams did not close within 10 seconds of the exit event from process 'C:\agent\_work\_tool\dotnet\dotnet.exe'. This may indicate a child process inherited the STDIO streams and has not yet exited.
2024-03-06T22:09:04.5283011Z Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting
2024-03-07T10:01:29.4560024Z ##[error]The Operation will be canceled. The next steps may not contain expected logs.
2024-03-07T10:01:29.5065729Z ##[error]The operation was canceled.
2024-03-07T10:01:29.5071108Z ##[section]Finishing: Install AzureSignTool

Repro steps

No response

KonstantinTyukalov commented 3 months ago

Hi @Lawls91 thanks for the reporting! We have higher priority issues now, but we'll get back to this one soon. How often do you encounter these hangups?

Lawls91 commented 3 months ago

Hey @KonstantinTyukalov sorry for the delay. It goes up and down in frequency. a couple weeks ago it was almost daily for example. At the moment, every few days maybe