Az.Storage version 6.1.1 cmdlets performance issue while using Federated Credential service connection.

shpengmsft commented 3 months ago

Description

This issue happened with Azure DevOps release pipeline.

This pipeline is using AzurePowershell@5 tasks to upload/update blobs on specific storage account. (These blobs are Python wheels or indexes.)

When the pipeline is using a service connection with Federated Credentials.

In the first 3~4 minutes, all these cmdlets (Set.AzContext, Set.AzCurrentStorageAccount) work fine. After that, we observed one of these cmdlets' executions time will be longer than 2.5 hours.

Detailed repro steps. Please contact me via internal email.

Issue script & Debug output

2024-06-20T23:13:35.7192921Z ##[section]Starting: Deploy packages/index
...
2024-06-20T23:14:16.8909540Z     Get-Blobs:: Getting blobs from storage '<***** Storage account name *****>', container 'wheels' with prefix '<***** Blob prefix *****>' and suffix '.html'
2024-06-20T23:14:16.8909932Z     Get-Blobs::Get-AzContext
2024-06-20T23:14:16.8910231Z     Get-Blobs:: Set-AzCurrentStorageAccount took 0.5183503 seconds
2024-06-20T23:15:49.0161572Z     Get-Blobs:: Get-AzStorageBlob took 92.72106 seconds
2024-06-20T23:15:49.0299267Z     Get-Blobs:: Found 165 blobs (excluding snapshots)
2024-06-20T23:15:49.0326070Z Creating backups for current target index
2024-06-20T23:15:49.0417778Z     Create-Backup:: Back-up container is not specified. Skipping 165 blobs
2024-06-20T23:15:49.0492424Z     Create-Snapshots:: Snapshots are not enabled. Skipping 165 blobs
2024-06-20T23:15:49.0524583Z 0
2024-06-20T23:15:49.0528154Z 1
2024-06-20T23:15:49.0570142Z Updating index
2024-06-20T23:15:49.0761010Z     Create-Index:: Creating index for <***** Storage account name *****>.azureedge.net endpoint
2024-06-20T23:15:49.0837207Z         Create-Index:: Looking for blobs in a location with id 'arm|<***** Storage account name *****>|<***** Storage account name *****>|<***** Subscription Name *****>|repo|<***** Blob prefix *****>'
2024-06-20T23:15:49.0862404Z             Get-Blobs:: Getting blobs from storage '<***** Storage account name *****>', container 'repo' with prefix '<***** Blob prefix *****>' and suffix '.whl'
2024-06-20T23:15:49.0890365Z             Get-Blobs::Get-AzContext
2024-06-20T23:15:49.6931297Z             Get-Blobs:: Set-AzCurrentStorageAccount took 0.5625513 seconds
2024-06-20T23:16:47.7566855Z             Get-Blobs:: Get-AzStorageBlob took 58.0590726 seconds
2024-06-20T23:16:47.7598845Z             Get-Blobs:: Found 10146 blobs (excluding snapshots)
2024-06-20T23:16:47.8496193Z         Create-Index:: New-AzStorageBlobSASToken generated SAS token for '<***** .whl ****>'
2024-06-20T23:16:47.8496193Z         Create-Index:: New-AzStorageBlobSASToken generated SAS token for '<***** .whl ****>'
2024-06-20T23:16:47.8496193Z         Create-Index:: New-AzStorageBlobSASToken generated SAS token for '<***** .whl ****>'
...
2024-06-20T23:22:47.3379612Z         Create-Index:: Looking for blobs in a location with id 'arm|<***** Storage account name *****>|<***** Storage account name *****>|<***** Subscription Name *****>|wheels|<***** Blob prefix *****>'
2024-06-20T23:22:47.3405929Z             Get-Blobs:: Getting blobs from storage '<***** Storage account name *****>', container 'wheels' with prefix '<***** Blob prefix *****>' and suffix '.whl'
2024-06-20T23:22:47.3435561Z             Get-Blobs::Get-AzContext
2024-06-21T01:47:55.7849622Z             Get-Blobs:: Set-AzCurrentStorageAccount took 8708.3301906 seconds
2024-06-21T01:49:24.1209139Z             Get-Blobs:: Get-AzStorageBlob took 88.3414451 seconds

Environment data

N/a please contact me via internal email.

Module versions

N/a please contact me via internal email.

Error output

N/a please contact me via internal email.

blueww commented 3 months ago

Hi @shpengmsft

How you get above log? (It looks not Azure Powershell cmdlets log.) Would you please share more information the issue template asked, so we can continue follow up the issue?

I will write a mail to you, you can share the info in mail.

BTW, the Azure DevOps pipeline AzurePowershell@5 task not owned by our team. You might need contact Azure Devops to get support on the AzurePowershell@5 task.

isra-fel commented 3 months ago

It's a complicated problem that involves in Azure Devops pipeline, Federated Identity, possible token cache persistance and MSAL. I'll update this issue once we've figure out the root cause.

shpengmsft commented 3 months ago

It's a complicated problem that involves in Azure Devops pipeline, Federated Identity, possible token cache persistance and MSAL. I'll update this issue once we've figure out the root cause.

I believe so. Thanks for reaching out via internal email. Let's use that to continue the investigation.

Azure / azure-powershell