Inconsistent performance issue in Azure Blob Storage Premium ZRS

[Murugaa19]

Library name and version

Azure.Storage.Blobs (12.15.0)

Query/Question

We are facing inconsistent performance issues repeatedly in Azure Blob Storage, particularly in the asynchronous copy blob process. Below is the code snippet we're using for copying:

We're initiating start copy requests that are queuing on the Azure storage without being processed. We aren't even receiving a server request ID to track the asynchronous operations, and without this ID, we're unable to cancel the requests that have already been initiated.

Sometimes, all requests are completed in milliseconds, while at other times, it takes more than 1 minute. Therefore, we included a workaround network timeout retry after 5 seconds. Now, most of the requests are taking 3-5 seconds and when retry it is getting completed in milliseconds.

Also, at that time, our service's CPU and memory usage are below 10%, so we don’t face any high load.

Provide a solution to resolve this performance issues.

Details:

Azure Storage Account	Premium (Performance), Zone-redundant storage (Replication)
SDK	Azure.Storage.Blobs (12.15.0)

Code used:

private static async Task CopyContainerAsync( BlobContainerClient sourceContainer, int pageSize, BlobContainerClient targetContainer, List excludedBlobs = default) { var i = 0;   await foreach (var batchItems in sourceContainer.GetBlobsAsync().AsPages(pageSizeHint: 100)) { var batchTasks = new List(); foreach (var item in batchItems.Values) { if (excludedBlobs?.Contains(item.Name) == true) { continue; }   BlobClient blob = sourceContainer.GetBlobClient(item.Name); BlobClient targetBlob = targetContainer.GetBlobClient(blob.Name); batchTasks.Add(targetBlob.StartCopyFromUriAsync(blob.Uri)); }   await Task.WhenAll(batchTasks).ConfigureAwait(false); } }

BlobClient blob = sourceContainer.GetBlobClient(item.Name); BlobClient targetBlob = targetContainer.GetBlobClient(blob.Name); await targetBlob.StartCopyFromUriAsync(blob.Uri, cancellationToken: cancellationToken)).configureawait(false);     services.AddAzureClients(builder => { builder.AddBlobServiceClient(EnvironmentVariables.AzureBlobStorage) .ConfigureOptions(options => { options.Retry.MaxRetries = 5; options.Retry.NetworkTimeout = TimeSpan.FromSeconds(5); }); });

Inconsistent Behavior in Copy Blob:

Environment

No response

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.