search

Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://learn.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.64k stars 2.84k forks source link

ServiceBusMessageBatch fails on case with multiple large messages #28421

Closed ZWMiller closed 1 year ago

ZWMiller commented 1 year ago

Describe the bug

I want to send a bunch of large JSON files to a ServiceBus Queue. Each JSON is ~800 kB. The subscription is premium, so I have 100MB limit in message size and the queue is set to 80GB max size. If I put all of these messages into a single JSON array and then send that array after compression to bytes with sender.send_messages(ServiceBusMessage(compressed_json)) it works fine. The messages in total are <40MB.

However, if I instead create a ServiceBusMessageBatch and add the messages 1 at a time with batch.add_message(ServiceBusMessage(comp_json)) and then try to send the batch with sender.send_messages(batch) I receive this error:

azure.servicebus.exceptions.ServiceBusError: Handler failed: Data set too large for a single message..

The message is still less than the 100MB limit on maximum message size.

I also tried creating the batch by making a list of messages and letting the sender auto-batch with:

batcher = []
for data_to_send in json_messages:
    batcher.append(gzip.compress(bytes(data_to_send.encode("utf-8"))))
sender.send_messages(batcher)

with the same error result. I believe there is some problem with the interplay between AMQP message size limits and the ServiceBus stated limit which only applying when ServiceBusMessageBatch is created instead of sending one single large message, but haven't been able to track down the exact problem other than ServiceBusMessageBatch seems to not subdivide properly, which AMQP tries to do to send.

I have also tried this without the compression and the same problem occurs, even when the message is just a string of valid JSON.

To Reproduce Steps to reproduce the behavior:

  1. Create a ServiceBusMessageBatch
  2. Add large objects (~1MB) into the batch with batch.add_message(ServiceBusMessage(large_object)). Make sure it's a few objects. I've seen this happen with as few as 2 objects
  3. Attempt to send the batch.

Alternate method

  1. Create an empty list.
  2. Add large objects (~1MB) into the list with batch.append(ServiceBusMessage(large_object)). Make sure it's a few objects. I've seen this happen with as few as 2 objects
  3. Attempt to send the list (which gets converted to a batch automatically)

Expected behavior Since the limit on my ServiceBus is 100MB, I expected the batch to send successfully

Additional context

Additional Trace:

2023-01-19 21:04:32,815 [ERROR] Unexpected error occurred (MessageContentTooLarge('Data set too large for a single message.')). Handler shutting down.
Traceback (most recent call last):
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 402, in _do_retryable_operation
    return operation(**kwargs)
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_servicebus_sender.py", line 276, in _send
    self._handler.send_message(message.message)
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/uamqp/client.py", line 729, in send_message
    batch = messages.gather()
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/uamqp/message.py", line 818, in gather
    raise errors.MessageContentTooLarge()
uamqp.errors.MessageContentTooLarge: Data set too large for a single message.
2023-01-19 21:04:32,816 [DEBUG] Destroying cMessageSender
2023-01-19 21:04:32,816 [INFO] Message sender b'sender-link-IDHERE' state changed from <MessageSenderState.Open: 3> to <MessageSenderState.Closing: 4> on connection: b'SBSender-IDHERE'
2023-01-19 21:04:32,816 [DEBUG] Destroying cLink
2023-01-19 21:04:32,816 [DEBUG] Deallocating 'CompositeValue'
2023-01-19 21:04:32,816 [DEBUG] Destroying 'CompositeValue'
2023-01-19 21:04:32,816 [DEBUG] Deallocating 'CompositeValue'
2023-01-19 21:04:32,817 [DEBUG] Destroying 'CompositeValue'
2023-01-19 21:04:32,817 [DEBUG] Deallocating cLink
2023-01-19 21:04:32,817 [DEBUG] Deallocating cMessageSender
2023-01-19 21:04:33,003 [DEBUG] CBS session pending.
2023-01-19 21:04:33,003 [DEBUG] Closing exclusive connection.
2023-01-19 21:04:33,003 [DEBUG] Unlocked connection b'SBSender-IDHERE' to close.
2023-01-19 21:04:33,003 [INFO] Shutting down connection b'SBSender-IDHERE'.
2023-01-19 21:04:33,003 [INFO] Shutting down CBS session on connection: b'SBSender-IDHERE'.
2023-01-19 21:04:33,003 [DEBUG] Unlocked CBS to close on connection: b'SBSender-IDHERE'.
2023-01-19 21:04:33,003 [DEBUG] Destroying CBSTokenAuth for connection b'SBSender-IDHERE'
2023-01-19 21:04:33,003 [INFO] Auth closed, destroying session on connection: b'SBSender-IDHERE'.
2023-01-19 21:04:33,003 [DEBUG] Destroying cSession
2023-01-19 21:04:33,003 [INFO] Finished shutting down CBS session on connection: b'SBSenderIDHERE.
2023-01-19 21:04:33,003 [DEBUG] Destroying Connection
2023-01-19 21:04:33,003 [INFO] Connection b'SBSender-IDHERE' state changed from <ConnectionState.OPENED: 9> to <ConnectionState.END: 13>
2023-01-19 21:04:33,004 [DEBUG] Destroying XIO
2023-01-19 21:04:33,004 [DEBUG] Destroying XIO
2023-01-19 21:04:33,004 [DEBUG] Destroying SASLMechanism
2023-01-19 21:04:33,004 [INFO] Connection shutdown complete b'SBSender-IDHERE'.
2023-01-19 21:04:33,005 [DEBUG] Deinitializing platform.
Traceback (most recent call last):
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 402, in _do_retryable_operation
    return operation(**kwargs)
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_servicebus_sender.py", line 276, in _send
    self._handler.send_message(message.message)
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/uamqp/client.py", line 729, in send_message
    batch = messages.gather()
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/uamqp/message.py", line 818, in gather
    raise errors.MessageContentTooLarge()
uamqp.errors.MessageContentTooLarge: Data set too large for a single message.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/azureuser/streaming-POCs/multimachine_experiments/json_compress_per_message_test.py", line 49, in <module>
    sender.send_messages(batcher)
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_servicebus_sender.py", line 450, in send_messages
    self._do_retryable_operation(
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 406, in _do_retryable_operation
    last_exception = self._handle_exception(exception)
  File "/home/azureuser/.local/share/virtualenvs/streaming-POCs-NyUVcXZ6/lib/python3.10/site-packages/azure/servicebus/_base_handler.py", line 353, in _handle_exception
    raise error
azure.servicebus.exceptions.ServiceBusError: Handler failed: Data set too large for a single message..
kashifkhan commented 1 year ago

Hi @ZWMiller ,

Thank for the issue. The documentation around this is certainly a bit difficult to find, but when messages are large batching is not supported by the service for performance reasons etc. It is recommended that you continue to send those messages individually.

ghost commented 1 year ago

Hi @ZWMiller. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

ZWMiller commented 1 year ago

Hey @kashifkhan . Thanks for the note and link. If I might make a recommendation, it would be helpful to have in the documentation what is considered a large message. I had read that section, but assumed that large message was 50-100MB. If I understand the source correctly, it looks like anything over 256kb is considered a large message for the purposes of batching. That information being in the docs would help solve for the issue.

Thanks again.

kashifkhan commented 1 year ago

Hi @ZWMiller, I agree with you around the confusion in that section. In the SDK docs we avoid putting down firm numbers since the limits/quotas can be changed by the service and we get limits etc. from the service during run time so that we can adjust on the fly.

On the SDK side we can link out to that document and I'll ask the service team if they can add some clarity around that particular section regarding what is a big message vs someone going over the limit of their subscription (standard versus premium).

The confusing aspect here itself is that the batch or the envelope in which the messages are delivered itself has a size limit and the messages have a size limit on the tier level.

ZWMiller commented 1 year ago

@kashifkhan That would be great. I think some acknowledgement in the docs/source code for ServiceBusMessageBatch that the limit is set by the handler, be it AMQP or other. That was the hardest part for me to track down when trying to figure out if I was doing something wrong or there was some hidden limitation. That implied size limit being tied to the handler abstracts a lot of the information around the behavior out of the class - which makes it very hard to track down.

Anyways, I'll stop making suggestions now. Appreciate your time. Cheers.