[BUG] BlobContainerClient.GetBlobs or GetBlobsAsync returns Response x-ms-client-request-id does not match the original expected request id

adeliab commented 1 year ago

Library name and version

Azure.Storage.Blobs 12.16.0

Describe the bug

Iterating through the result of BlobContainerClient.GetBlobsAsync or GetBlobs throws Azure.RequestFailedException with message: "Response x-ms-client-request-id 'xxx' does not match the original expected request id, 'yyy'."

This only happens in subsequent runs, not in initial run. In my test application, doing either these two will 'fix' the issue:

Using a new download uri
Upgrading/ downgrading the Azure.Storage.Blobs version (I tried 12.15.1). The same error happens there and I upgraded back to 12.16.1 and it goes ok the first run, then it happens again
Waiting for a few minutes

The download uri is a SAS uri with custom domain. I tried setting the clientOptions.Retry.Mode to exponential and maxretry=5 but it doesnt solve the issue

Expected behavior

Blobs retrieved successfully

Actual behavior

It throws Azure.RequestFailedException with message: "Response x-ms-client-request-id 'xxx' does not match the original expected request id, 'yyy'."

Reproduction Steps

            var uri = new Uri(downloadUri);
            var containerClient = new BlobContainerClient(uri);
            var blobs = containerClient.GetBlobsAsync();

            var blobClients = new Dictionary<string, BlobClient>();
            await foreach (var blob in blobs)
            {
                var blobClient = containerClient.GetBlobClient(blob.Name);
                blobClients.Add(blob.Name, blobClient);
            }
            return blobClients;

The line await foreach (var blob in blobs) throws Azure.RequestFailedException with message: "Response x-ms-client-request-id 'xxx' does not match the original expected request id, 'yyy'." Stack trace:

at Azure.Storage.StorageRequestValidationPipelinePolicy.OnReceivedResponse(HttpMessage message)
   at Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.<InnerProcessAsync>d__5.MoveNext()
   at Azure.Core.Pipeline.RedirectPolicy.<ProcessAsync>d__7.MoveNext()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
   at Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__5.MoveNext()
   at Azure.Storage.Blobs.ContainerRestClient.<ListBlobFlatSegmentAsync>d__56.MoveNext()
   at Azure.Storage.Blobs.BlobContainerClient.<GetBlobsInternal>d__77.MoveNext()
   at Azure.Storage.Blobs.Models.GetBlobsAsyncCollection.<GetNextPageAsync>d__5.MoveNext()
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at Azure.Storage.StorageCollectionEnumerator`1.StorageAsyncPageable.<GetAsyncEnumerator>d__5.MoveNext()
   at Azure.Storage.StorageCollectionEnumerator`1.StorageAsyncPageable.<GetAsyncEnumerator>d__5.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
   at System.Runtime.CompilerServices.ValueTaskAwaiter`1.GetResult()
   at TestConsole.MessageHandler.<GetBlobClientsAsync>d__5.MoveNext() in C:\projects\Packages\TestConsole\MessageHandler.cs:line 65
   at TestConsole.MessageHandler.<GetBlobClientsAsync>d__5.MoveNext() in C:\projects\Packages\TestConsole\MessageHandler.cs:line 65
   at Program.<<Main>$>d__0.MoveNext() in C:\projects\Packages\TestConsole\Program.cs:line 10

The downloadUri is a SAS uri with custom domain.

Environment

.NET SDK: Version: 7.0.202 Commit: 6c74320bc3

Runtime Environment: OS Name: Windows OS Version: 10.0.19044 OS Platform: Windows RID: win10-x64 Base Path: C:\Program Files\dotnet\sdk\7.0.202\

Host: Version: 7.0.4 Architecture: x64 Commit: 0a396acafe

The application is running in .net 6

github-actions[bot] commented 1 year ago

Thank you for your feedback. This has been routed to the support team for assistance.

adeliab commented 1 year ago

We think that the issue happens when we perform an operation twice or more times with the same client (either blobcontainerclient or blobclient). Scenarios that would reproduce this error:

Creating a BlobContainerClient with the same URI and performing .GetBlobs(). Fails during iteration of the result. This succeeds the first time and then fails even after the application is stopped and executed again
Creating a BlobContainerClient, call.GetBlobs(), do .Count() and then iterate the blobs.
Creating a BlobContainerClient, do 2 GetBlobClient with the same blob name

The custom domain seems to have an impact since we didn't have this issue before using custom domain

github-actions[bot] commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.

amnguye commented 1 year ago

Sounds like there's a proxy cache in between altering the x-ms-client-request-id. I highly recommend you disable the cache as it's altering the requests sent by the Storage SDK.

adeliab commented 1 year ago

@amnguye thanks for your reply. we expect so too, but how do we disable this cache? can you provide more instructions for it? Thanks

amnguye commented 1 year ago

Sorry for the late reply. A cache could be implemented in many places (e.g. firewall, network, client-side). I can't provide you specific instructions on how to disable it since it's not SDK related.

cdawnjodata commented 8 months ago

Is there any update on this? Or any way to get around this issue? I really need to be able to list files in a storage account, and everything I do inside a C# azure function gets me to the same error.

Azure / azure-sdk-for-net