search

Azure / azure-sdk-for-net

This repository is for active development of the Azure SDK for .NET. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/dotnet/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-net.
MIT License
5.48k stars 4.81k forks source link

[QUERY] Understanding copy status when blob is deleted after (?during) async copy operation within same storage account #46385

Open potomato opened 1 month ago

potomato commented 1 month ago

Library name and version

Azure.Storage.Blobs 12.21.2

Query/Question

Hi,

We are copying blobs from one container to another within the same storage account, as we would like it to be an atomic operation (we use Event Hub to push the Blob Created message to a storage queue which is consumed by an Azure function to process the blob).

We are using this code:

var blobServiceClient = new BlobServiceClient(connectionString);
var sourceBlobContainerClient = blobServiceClient.GetBlobContainerClient(fileDetails.TemporaryContainerName);
var sourceBlobClient = sourceBlobContainerClient.GetBlobClient(fileDetails.FileName);
var destBlobContainerClient = blobServiceClient.GetBlobContainerClient(fileDetails.ContainerName);
var destBlobClient = destBlobContainerClient.GetBlobClient(fileDetails.FileName);

var status = await destBlobClient.StartCopyFromUriAsync(sourceBlobClient.Uri);
await status.WaitForCompletionAsync();

We are finding that the blob is collected and deleted before the final WaitForCompletionAsync, so this receives 404 and throws an exception. This isn't wrong but makes it hard to understand what has happened. Did the copy complete successfully?

These are the sanitised blob storage logs: 3:04:05.424 PM,Writer,/storageaccount/tempcontainer/blobname,PutBlob,201,Success 3:04:05.435 PM,Writer,/storageaccount/finalcontainer/blobname,CopyBlob,202,Success 3:04:05.435 PM,Writer,https://storageaccount.blob.core.windows.net/tempcontainer/blobname,CopyBlobSource,202,Success 3:04:05.435 PM,Writer,/storageaccount/finalcontainer/blobname,CopyBlobDestination,202,Success 3:04:05.707 PM,Consumer,/storageaccount/finalcontainer/blobname,GetBlobProperties,200,Success 3:04:05.712 PM,Consumer,/storageaccount/finalcontainer/blobname,GetBlobProperties,200,Success 3:04:05.719 PM,Consumer,/storageaccount/finalcontainer/blobname,GetBlob,200,Success 3:04:05.770 PM,Consumer,/storageaccount/finalcontainer/blobname,GetBlobProperties,200,Success 3:04:05.777 PM,Consumer,/storageaccount/finalcontainer/blobname,GetBlob,200,Success 3:04:05.823 PM,Consumer,/storageaccount/finalcontainer/blobname,GetBlobProperties,200,Success 3:04:05.828 PM,Consumer,/storageaccount/finalcontainer/blobname,DeleteBlob,202,Success 3:04:06.405 PM,Writer,/storageaccount/finalcontainer/blobname,GetBlobProperties,404,BlobNotFound

The last two lines show the consumer deleting the blob before the writer has done its final check as part of WaitForCompletionAsync.

How do I interpret this in my client code? Does a 404 in WaitForCompletionAsync guarantee that the copy completed successfully and that the file was subsequently deleted? If not how do I tell the difference between this condition and 'some other error condition where the blob disappeared'.

Environment

Azure Function, Isolated model on .net 8

github-actions[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage.