search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.43k stars 2.59k forks source link

[REGRESSION]: AzureFileCopy@6 failing to load Az.Accounts v3.0.0 module #19910

Open MSylvia opened 1 month ago

MSylvia commented 1 month ago

New issue checklist

Task name

AzureFileCopy

Breaking task version

6.240.1

Last working task version

6.239.x

Regression Description

We have a pipeline that is using AzureFileCopy@6 with a managed identity. It has been in place for about 2 weeks and has been running without issue. On Tuesday the task started failing with the only visible change in the logs being the Az.Accounts module updating from v2.9.1 to v3.0.0.

When it fails, we get this error:

Failed to perform Auto-login: PSContextCredentialGet-AzAccessToken: -REDACTED- 'Get-AzAccessToken' command was found in the module 'Az.Accounts', but the module could not be loaded. For more information, run 'Import-Module Az.Accounts'.

In failed runs the listed Az.Accounts module version is always v3.0.0. 

Imported module 'Az.Accounts', version: 3.0.0

Runs that executed earlier which passed have a version of v2.9.1.

Imported module 'Az.Accounts', version: 2.9.1

Cross contamination of jobs is not possible because a fresh machine with the exact same configuration is used for each run and that configuration has not changed.

I believe it may have been caused by this change from AzureRM to Az: https://github.com/microsoft/azure-pipelines-tasks/commit/f5b22d9c64118dc809b217a09e4e5d8599939429

Example runs:

Pass: v2.9.1 https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9600629&view=logs&j=4736f78b-748e-51a8-c3dc-ea7706a06c3d&t=f124d820-a316-55db-635e-ba56d272505a Fail: v3.0.0 https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9613411&view=logs&j=4736f78b-748e-51a8-c3dc-ea7706a06c3d&t=f124d820-a316-55db-635e-ba56d272505a

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

Starting: Upload to Storage
==============================================================================
Task         : Azure file copy
Description  : Copy files to Azure Blob Storage or virtual machines
Version      : 6.240.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
==============================================================================
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\3.0.0\Az.Accounts.psd1 -Global -PassThru -Force
##[warning]Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide
Imported module 'Az.Accounts', version: 3.0.0
Update-AzConfig -DisplayBreakingChangeWarning False -AppliesTo Az
Enable-AzureRmAlias -Scope Process
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Resources\1.13.0\Az.Resources.psd1 -Global -PassThru -Force
Imported module 'Az.Resources', version: 1.13.0
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Storage\1.14.0\Az.Storage.psd1 -Global -PassThru -Force
Imported module 'Az.Storage', version: 1.14.0
Uninstall-AzureRm
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Connect-AzAccount 

......

Failed to perform Auto-login: PSContextCredentialGet-AzAccessToken: -REDACTED- 'Get-AzAccessToken' command was found in the module 'Az.Accounts', but the module could not be loaded. For more information, run 'Import-Module Az.Accounts'.

Full task logs with system.debug enabled

We don't have full task logs with system.debug enabled.

Repro steps

- task: AzureFileCopy@6
  displayName: 'Azure File Copy'
  inputs:
    SourcePath: '${{ parameters.BuildPackages }}'
    azureSubscription: ${{ parameters.AzureSubscription }}
    Destination: AzureBlob
    storage: ${{ parameters.AzureStorage }}
    ContainerName: ${{ parameters.AzureContainerName }}
    BlobPrefix: ${{ parameters.AzureUploadLocation }}
  condition: ${{ parameters.condition }}
v-snalawade commented 1 month ago

@MSylvia - Could you please run the pipeline with debug log enabled and share the logs.

MSylvia commented 1 month ago

@v-snalawade To test if the warning Both Az and AzureRM modules were detected on this machine was the cause, I ran a different pipeline on two modified images and got the same error.

With warning (Both Az and AzureRM on machine):

Without warning (Only Az):

Both runs in this pipeline have debug logs enabled.

v-snalawade commented 1 month ago

@MSylvia - I dont see debug logs enabled, please add variable system.debug to true and run pipeline again.

MSylvia commented 1 month ago

@v-snalawade Here is a new run with debug logs enabled.

With warning (Both Az and AzureRM on machine):

Without warning (Only Az):

v-snalawade commented 1 month ago

@MSylvia - I went through logs and these are not debug logs. please add variable system.debug to true in pipeline variables and run pipeline again, you will get the detailed logs.

MSylvia commented 1 month ago

@v-snalawade Here is a new run with debug logs actually enabled this time. I mistakenly thought that enabling diagnostics also set debug to true before.

With warning (Both Az and AzureRM on machine):

Without warning (Only Az):

jonathanjyi commented 1 month ago

I'm running into the same issue: Breaking task version 6.240.1 6.239.x

It seems like both 240 and 239 are broken when using Az.Accounts v3.0.0

jonathanjyi commented 1 month ago

Debug Logs here: https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=1

MSylvia commented 1 month ago

@DmitriiBobreshev I see you made some changes recently to the Azure File Copy tasks. Do you know what might cause the issues we are seeing?

v-snalawade commented 1 month ago

Debug Logs here: https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=1

Thanks for the logs, will check and keep you posted on updates

jonathanjyi commented 1 month ago

Thanks, are there any workarounds we can employ in the meantime?

From: v-snalawade @.> Sent: Wednesday, June 5, 2024 7:06 AM To: microsoft/azure-pipelines-tasks @.> Cc: Comment @.> Subject: Re: [microsoft/azure-pipelines-tasks] [REGRESSION]: @. failing to load Az.Accounts v3.0.0 module (Issue #19910)

Debug Logs here: https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=1

Thanks for the logs, will check and keep you posted on updates

- Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/19910#issuecomment-2149666752 or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYEROG4772D5KSFKUTZZPT3ZF35JDBFKMF2HI4TJMJ2XIZLTTCBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEYTKNBYGE2DAMJRURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2SMZZGEYDQOBSGMYKI3TBNVS2S2DBONPWYYLCMVWIFJLWMFWHKZNKGEZDQMBZGQZTGNRSURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UMJWG43TCMBRHAYDDJDOMFWWLKLIMFZV63DBMJSWZAVFOZQWY5LFVI2TQMJXG4YTONBQGWSG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44XQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKBSG43DGMJWG4ZYFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDGMJWGMYTINJSGSBKI5DZOBS2K3DBMJSWZJLWMFWHKZNJGE2TIOBRGQYDCMMCUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMZZGEYDQOBSGMYIFJDUPFYGLJLMMFRGK3FFOZQWY5LFVIYTEOBQHE2DGMZWGKBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKGE3DONZRGAYTQMBRQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKRVHAYTONZRG42DANNHORZGSZ3HMVZKMY3SMVQXIZI. You are receiving this email because you commented on the thread.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

DmitriiBobreshev commented 1 month ago

Hi @MSylvia, @jonathanjyi, Most probably the changes not causing the issue as the changes FF for AzureRM migration were ON for a while for both organizations. I see that both pools are using the same image and suspect that there is some inconsistency between PowerShell where the modules are enabling and pwsh which is used for Get-AzAccessToken CmdLet inside AzCopy.exe. Moreover, the other CmdLets as Connect-AzAccount or Set-AzContext(both provided by Az.Accounts) succeeded, which tells us that the module was loaded correctly for PowerShell session but probably wasn't loaded for pwsh. Is it possible to try another a pool with different image to check will it works for you?

jonathanjyi commented 1 month ago

Thanks Dmitrii, this was very helpful. I switched images, and it does seem to work.

From: Dmitrii Bobreshev @.> Sent: Wednesday, June 5, 2024 8:36 AM To: microsoft/azure-pipelines-tasks @.> Cc: Mention @.>; Comment @.> Subject: Re: [microsoft/azure-pipelines-tasks] [REGRESSION]: @.*** failing to load Az.Accounts v3.0.0 module (Issue #19910)

Hi @MSylviahttps://github.com/MSylvia, @jonathanjyihttps://github.com/jonathanjyi, Most probably the changes not causing the issue as the changes FF for AzureRM migration were ON for a while for both organizations. I see that both pools are using the same image and suspect that there is some inconsistency between PowerShell where the modules are enabling and pwsh which is used for Get-AzAccessTokenhttps://github.com/Azure/azure-storage-azcopy/blob/main/common/azure_ps_context_credential.go#L125C10-L125C27 CmdLet inside AzCopy.exe. Moreover, the other CmdLets as Connect-AzAccounthttps://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=2936 or Set-AzContexthttps://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=2957(both provided by Az.Accounts) succeeded, which tells us that the module was loaded correctly for PowerShell session but probably wasn't loaded for pwsh. Is it possible to try another a pool with different image to check will it works for you?

- Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-tasks/issues/19910#issuecomment-2149958912 or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYEROG73C5ZZMFTY6U45ITDZF4H4ZBFKMF2HI4TJMJ2XIZLTTCBKK5TBNR2WLJDUOJ2WLJDOMFWWLO3UNBZGKYLEL5YGC4TUNFRWS4DBNZ2F6YLDORUXM2LUPGBKK5TBNR2WLJDUOJ2WLJDOMFWWLLTXMF2GG2C7MFRXI2LWNF2HTAVFOZQWY5LFUVUXG43VMWSG4YLNMWVXI2DSMVQWIX3UPFYGLAVFOZQWY5LFVEYTKNBYGE2DAMJRURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2SMZZGEYDQOBSGMYKI3TBNVS2S2DBONPWYYLCMVWIFJLWMFWHKZNKGEZDQMBZGQZTGNRSURXGC3LFVFUGC427NRQWEZLMQKSXMYLMOVS2UMJWG43TCMBRHAYDDJDOMFWWLKLIMFZV63DBMJSWZAVFOZQWY5LFVI2TQMJXG4YTONBQGWSG4YLNMWUWQYLTL5WGCYTFNSWHG5LCNJSWG5C7OR4XAZNMJFZXG5LFINXW23LFNZ2KM5DPOBUWG44XQKSHI6LQMWVHEZLQN5ZWS5DPOJ42K5TBNR2WLKBSG43DGMJWG4ZYFJDUPFYGLJLJONZXKZNFOZQWY5LFVIZDGMJWGMYTINJSGSBKI5DZOBS2K3DBMJSWZJLWMFWHKZNJGE2TIOBRGQYDCMMCUR2HS4DFUVWGCYTFNSSXMYLMOVS2SMZZGEYDQOBSGMYIFJDUPFYGLJLMMFRGK3FFOZQWY5LFVIYTEOBQHE2DGMZWGKBKI5DZOBS2K3DBMJSWZJLWMFWHKZNKGE3DONZRGAYTQMBRQKSHI6LQMWSWYYLCMVWKK5TBNR2WLKRVHAYTONZRG42DANNHORZGSZ3HMVZKMY3SMVQXIZI. You are receiving this email because you were mentioned.

Triage notifications on the go with GitHub Mobile for iOShttps://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Androidhttps://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

MSylvia commented 1 month ago

@DmitriiBobreshev While switching pools / images worked for @jonathanjyi there still may be a problem because that pool is loading the older version of Az.Accounts.

Imported module 'Az.Accounts', version: 2.19.0

https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28477100&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=92

If/when that pool updates it could still be a problem.

DevTestUser123 commented 4 weeks ago

Hi all any update on this? getting same issue

MSylvia commented 4 days ago

Since pinning the task version also fails, it seems like it could be related to this issue: https://github.com/microsoft/azure-pipelines-tasks/issues/16505

v-schhabra commented 2 days ago

20079 This is the fix created for the above issue. Will update here once the fixes are deployed to all the rings.