rebus-org / Rebus.AzureBlobs

:bus: Azure Blobs-based databus storage for Rebus
https://mookid.dk/category/rebus
Other
3 stars 5 forks source link

Could not upload data to blob - The argument must not be empty string #5

Closed Nisden closed 10 months ago

Nisden commented 10 months ago

Enabling SendBigMessagesAsAttachments together with StoreInBlobStorage throws an exception that says "The argument must not be empty string"

Our configuration is pretty basic

var dataBusOptions = d.StoreInBlobStorage(configuration.GetConnectionString("AzureStorage"), storageOptions.DataBusContainer);
d.SendBigMessagesAsAttachments(0); // When we approach the limit of Azure Storage Queues we want to transfer to blob storage
d.EnableEncryption(); // Encrypted while stored

if (!storageOptions.AllowAutoCreate)
    dataBusOptions.DoNotCreateContainer();

Full exception

Rebus.Exceptions.RebusApplicationException: Could not create (automatic claim check) attachment for outgoing message with ID 83b2a3fe-a812-4874-9da2-41b747691db5
 ---> System.IO.IOException: Could not upload data to blob named 'data-5c7c7972-b94c-4b53-a249-225b2e69f627.dat' in the 'databus' container
 ---> Microsoft.Azure.Storage.StorageException: The argument must not be empty string.
 ---> System.ArgumentException: The argument must not be empty string.
   at Microsoft.Azure.Storage.Shared.Protocol.HttpRequestMessageFactory.AddMetadata(StorageRequestMessage request, String name, String value)
   at Microsoft.Azure.Storage.Shared.Protocol.HttpRequestMessageFactory.AddMetadata(StorageRequestMessage request, IDictionary`2 metadata)
   at Microsoft.Azure.Storage.Blob.Protocol.BlobHttpRequestMessageFactory.AddMetadata(StorageRequestMessage request, IDictionary`2 metadata)
   at Microsoft.Azure.Storage.Blob.CloudBlockBlob.<>c__DisplayClass145_0.<PutBlobImpl>b__1(RESTCommand`1 cmd, Uri uri, UriQueryBuilder builder, HttpContent cnt, Nullable`1 serverTimeout, OperationContext ctx)
   at Microsoft.Azure.Storage.Core.Executor.Executor.ProcessStartOfRequest[T](ExecutionState`1 executionState, String startLogMessage, CancellationTokenSource timeoutTokenSource)
   at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   --- End of inner exception stack trace ---
   at Microsoft.Azure.Storage.Core.Executor.Executor.ExecuteAsync[T](RESTCommand`1 cmd, IRetryPolicy policy, OperationContext operationContext, CancellationToken token)
   at Microsoft.Azure.Storage.Blob.CloudBlockBlob.UploadFromStreamAsyncHelper(Stream source, Nullable`1 length, AccessCondition accessCondition, BlobRequestOptions options, OperationContext operationContext, AggregatingProgressIncrementer progressIncrementer, CancellationToken cancellationToken)
   at Rebus.AzureBlobs.DataBus.AzureBlobsDataBusStorage.Save(String id, Stream source, Dictionary`2 metadata)
Request Information
RequestID:
RequestDate:
StatusMessage:
ErrorCode:

   --- End of inner exception stack trace ---
   at Rebus.AzureBlobs.DataBus.AzureBlobsDataBusStorage.Save(String id, Stream source, Dictionary`2 metadata)
   at Rebus.Encryption.EncryptingDataBusStorageDecorator.Save(String id, Stream source, Dictionary`2 metadata)
   at Rebus.DataBus.DefaultDataBus.CreateAttachment(Stream source, Dictionary`2 optionalMetadata)
   at Rebus.DataBus.ClaimCheck.DehydrateOutgoingMessageStep.DehydrateTransportMessage(OutgoingStepContext context, TransportMessage transportMessage)
   --- End of inner exception stack trace ---
   at Rebus.DataBus.ClaimCheck.DehydrateOutgoingMessageStep.DehydrateTransportMessage(OutgoingStepContext context, TransportMessage transportMessage)
   at Rebus.DataBus.ClaimCheck.DehydrateOutgoingMessageStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.Encryption.EncryptMessagesOutgoingStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.Pipeline.Send.SerializeOutgoingMessageStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.Pipeline.Send.AutoHeadersOutgoingStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.Pipeline.Send.FlowCorrelationIdStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.Pipeline.Send.AssignDefaultHeadersStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.ServiceProvider.ServiceProviderProviderStep.Process(OutgoingStepContext context, Func`1 next)
   at Rebus.Bus.RebusBus.SendUsingTransactionContext(IEnumerable`1 destinationAddresses, Message logicalMessage, ITransactionContext transactionContext)
   at Rebus.Bus.RebusBus.InnerSend(IEnumerable`1 destinationAddresses, Message logicalMessage)
   at Rebus.Bus.RebusBus.Send(Object commandMessage, IDictionary`2 optionalHeaders)
Nisden commented 10 months ago

Might have found the bug (And will be testing it very soon), I think its because my custom encryption have an empty IV.

mookid8000 commented 10 months ago

I've tried to reproduce it, but without success.

The test is here: https://github.com/rebus-org/Rebus.AzureBlobs/blob/master/Rebus.AzureBlobs.Tests/Bugs/ReproduceDataBusNullReferenceException.cs

Can you see if there's anything different in your code?

Nisden commented 10 months ago

Just confirmed it, my issue was my custom encryption based on a DataProtectionProvider

internal sealed class RebusDataProtectionEncryption : IEncryptor
{
    private readonly IDataProtector dataProtector;

    public string ContentEncryptionValue => "dataprotection";

    public RebusDataProtectionEncryption(IDataProtectionProvider dataProtectionProvider)
    {
        dataProtector = dataProtectionProvider.CreateProtector("Rebus");
    }

    public byte[] Decrypt(EncryptedData encryptedData)
    {
        return dataProtector.Unprotect(encryptedData.Bytes);
    }

    public EncryptedData Encrypt(byte[] bytes)
    {
        return new EncryptedData(dataProtector.Protect(bytes), Array.Empty<byte>());
    }
}

Having added an "dummy" IV it works now, instead of Array.Empty