The total payload size of a message batch is not handled automatically

[dietermijle]

The documentation of SendMessageBatchRequest says the following:

The maximum allowed individual message size and the maximum total payload size (the sum of the individual lengths of all of the batched messages) are both 256 KB (262,144 bytes).

This lib seems to handle the max allowed individual message size properly, but not the max total payload size.

public override async Task<SendMessageBatchResponse> SendMessageBatchAsync(SendMessageBatchRequest sendMessageBatchRequest, CancellationToken cancellationToken = default(CancellationToken))
{
    if (sendMessageBatchRequest == null)
    {
        throw new AmazonClientException("sendMessageBatch cannot be null");
    }

    if (!clientConfiguration.IsLargePayloadSupportEnabled)
    {
        return await base.SendMessageBatchAsync(sendMessageBatchRequest, cancellationToken).ConfigureAwait(false);
    }

    for (var i = 0; i < sendMessageBatchRequest.Entries.Count; i++)
    {
        if (clientConfiguration.AlwaysThroughS3 || IsLarge(sendMessageBatchRequest.Entries[i]))
        {
           sendMessageBatchRequest.Entries[i] = await StoreMessageInS3Async(sendMessageBatchRequest.Entries[i], cancellationToken);
        }
    }

    return await base.SendMessageBatchAsync(sendMessageBatchRequest, cancellationToken);
}

For example the SendMessageBatchRequest contains 2 entries both of size 200KB. The fallback to S3 will not be handled for either of these entries when using the default threshold of 256KB, but the SendMessageBatchAsync will fail as the total payload is 400KB which exceeds the max allowed size of 256KB.

[dietermijle]

I did some further investigation and found that this issue occurs when

• every message separately does not exceed the max size limit (256KB) AND
• the total payload in the batch (all messages together) exceed the max limit (256KB) AND
• messages are sent within the handle of another message (Rebus is creating a TransactionContext in this case) which is creating batched messaging

I'm using the MessageBatchSize of Amazon.SQS library in combination with the MessageSizeThreshold of this lib to make sure that the max payload size is never reached.