robinrodricks / FluentStorage

A polycloud .NET cloud storage abstraction layer. Provides Blob storage (AWS S3, GCP, FTP, SFTP, Azure Blob/File/Event Hub/Data Lake) and Messaging (AWS SQS, Azure Queue/ServiceBus). Supports .NET 5+ and .NET Standard 2.0+. Pure C#.
MIT License
272 stars 34 forks source link

FluentStorage.Azure.Blobs.WriteAsync does not support "append=true" #79

Open gentledepp opened 2 days ago

gentledepp commented 2 days ago

Even though the method signature suggests it, AzureBlobStorages

public async Task WriteAsync(
      string fullPath,
      Stream dataStream,
      bool append = false,
      CancellationToken cancellationToken = default (CancellationToken))

setting append to true still just overwrites any preexisting blob.

This is because it uses BlockBlobClient.UploadAsync internally, where the documentation clearly states:

The UploadAsync(Stream, BlobUploadOptions, CancellationToken) operation overwrites the contents of the blob, creating a new block blob if none exists. Overwriting an existing block blob replaces any existing metadata on the blob.

gentledepp commented 2 days ago

Note. Naively using the OpenWriteAsync API that - by its signature - supports appending to an existing blockblob like so:

    public async Task WriteAsync(string fullPath, Stream dataStream,
           bool append = false, CancellationToken cancellationToken = default) {
            GenericValidation.CheckBlobFullPath(fullPath);

            if (dataStream == null)
                throw new ArgumentNullException(nameof(dataStream));

            (BlobContainerClient container, string path) = await GetPartsAsync(fullPath, true).ConfigureAwait(false);

            BlockBlobClient client = container.GetBlockBlobClient(path);

            try {
                var targetStream = await client
                    .OpenWriteAsync(!append, new BlockBlobOpenWriteOptions(), cancellationToken: cancellationToken)
                    .ConfigureAwait(false);
                await dataStream.CopyToAsync(targetStream, 4096, cancellationToken).ConfigureAwait(false);
                // await client.UploadAsync(
                //    new StorageSourceStream(dataStream),
                //    cancellationToken: cancellationToken).ConfigureAwait(false);
            }
            catch (RequestFailedException ex) when (ex.ErrorCode == "OperationNotAllowedInCurrentState") {
                //happens when trying to write to a non-file object i.e. folder
            }
        }

throws an Exception

System.ArgumentException : BlockBlobClient.OpenWrite only supports overwriting bei Azure.Storage.Blobs.Specialized.BlockBlobClient.d__58.MoveNext()

So this is a dead end too...

gentledepp commented 2 days ago

Since we have potentially huge files we need to stream to Azure Blobstorage, the only viable alternative I found was to extend the IAzureBlobStorage interface with a new method Task<Stream> OpenWriteAsync(string fullPath, CancellationToken cancellationToken = default);