search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.49k stars 2.61k forks source link

[BUG]: Azure File Copy Task (Version 6) is breaking on pipeline using Self hosted agent with error 'The term 'azcopy' is not recognized. #20003

Closed Varaprasad91 closed 2 months ago

Varaprasad91 commented 3 months ago

New issue checklist

Task name

Azure file copy

Task version

6.241.0

Issue Description

We recently upgraded the Azure DevOps (ADO) service connection from using an Azure Resource Manager service principal (automatic) to Azure Resource Manager using Workload Identity federation with OpenID Connect (automatic). But, we got an issue with Azure file copy task (Task version 6) when running it on pipeline with self-hosted agent. The error message received was: 'The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or operable program. Please check the spelling of the name, or verify that the path is correct and try again.' Also, the task works correctly when using an Azure agent, but fails on the self-hosted agent.

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

windows

Relevant log output

[debug]Script stack trace:
[debug]at ThrowError, C:\Agent\vsts-agent-win-x64-2.182.1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.241.0\Utility.ps1: line 121
[debug]at Upload-FilesToAzureContainer, C:\Agent\vsts-agent-win-x64-2.182.1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.241.0\Utility.ps1: line 272
[debug]at <ScriptBlock>, C:\Agent\vsts-agent-win-x64-2.182.1\_work\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.241.0\AzureFileCopy.ps1: line 181
[debug]at <ScriptBlock>, <No file>: line 1
[debug]at <ScriptBlock>, <No file>: line 22
[debug]at <ScriptBlock>, <No file>: line 18
[debug]at <ScriptBlock>, <No file>: line 1
[debug]Exception:
System.Management.Automation.RuntimeException: Upload to container: 'sitemap-generator' in storage account: 'sitemapextractor' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
2024-06-13T07:48:44.0282744Z ##[error]Upload to container: 'sitemap-generator' in storage account: 'sitemapextractor' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
[debug]Processed: ##vso[task.logissue type=error;source=TaskInternal]Upload to container: 'sitemap-generator' in storage account: 'sitemapextractor' with blob prefix: '' failed with error: 'AzCopy.exe exited with non-zero exit code while uploading files to blob storage.' For more info please refer to https://aka.ms/azurefilecopyreadme
[debug]Processed: ##vso[task.complete result=Failed]
[section]Finishing: AzureBlob File Copy

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION]

Repro steps

No response

### Tasks
v-schhabra commented 3 months ago

Hi @Varaprasad91 Could you please share us the complete logs of the pipeline by adding variable system.debug to "true".

Varaprasad91 commented 3 months ago

error logs1.txt @v-schhabra could you please check this

v-schhabra commented 3 months ago

Hi @Varaprasad91 Thanks for sharing the logs. I hope @v-snalawade already connected with you reg this issue and she has started investigation.

tomaeda commented 3 months ago

Hi, @v-snalawade I'm having trouble with the same issue. Are there any updates? Auto-login appears to fail because azcopy.exe isn't available for the oauth token task has retrieved, even though task has running Connect-AzAccount. Same issue as #19910 ?

v-schhabra commented 3 months ago

Hi, @v-snalawade I'm having trouble with the same issue. Are there any updates? Auto-login appears to fail because azcopy.exe isn't available for the oauth token task has retrieved, even though task has running Connect-AzAccount. Same issue as #19910 ?

We are already investigating on this issue. Will share the latest updates here. Could you please share us the complete logs of the pipeline by adding variable system.debug to "true".

v-snalawade commented 3 months ago

@tomaeda - Can you check if this suggestion resolves the issue

tomaeda commented 3 months ago

Thank you for your response.

@v-schhabra I'm sorry, the logs I'm seeing are provided by the end user. I can't share it here. I haven't been able to reproduce this issue, so I don't have my own logs. However, the information from line 5599 onwards in the error.logs1.txt shared by Varaprasad91 is consistent with the logs I am seeing, except for specific information such as the storage account name.

@v-snalawade End user can work around the issue with a different pool. The issue only occurs with the Self-hosted agent pool, not with MS-hosted pool. But end user wants to use Self-hosted, so I'd like to know how to resolve the inconsistency between powershell and pwsh. (If this is same issue)

v-schhabra commented 3 months ago

Thank you for your response.

@v-schhabra I'm sorry, the logs I'm seeing are provided by the end user. I can't share it here. I haven't been able to reproduce this issue, so I don't have my own logs. However, the information from line 5599 onwards in the error.logs1.txt shared by Varaprasad91 is consistent with the logs I am seeing, except for specific information such as the storage account name.

@v-snalawade End user can work around the issue with a different pool. The issue only occurs with the Self-hosted agent pool, not with MS-hosted pool. But end user wants to use Self-hosted, so I'd like to know how to resolve the inconsistency between powershell and pwsh. (If this is same issue)

If it is fine for you please share the logs at v-schhabra@microsoft.com

tomaeda commented 3 months ago

I can't upload the logs I'm looking at. I'm trying to reproduce it in my environment so that I can provide the logs, but unfortunately for now I keep failing to reproduce.

v-schhabra commented 2 months ago

20117

Did fixes for the above issue and deployments are completed to all the rings.