Question about calculating message size

[zoe-mcelderry]

This is more of a question / feature request than an issue...

Is there any support for calculating the size of a message?

I found this page https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct that explains what contributes to the message size, but it is not clear how to calculate the size of each of the system properties? I can see that some of them are accessible from iothub_message.h, but it doesn't seem to provide access to all of them. Ideally it would be nice to either have an upper bound on how large the system properties can be or have a method made available to calculate the size of them -- do either of these exist?

I find the feedback when the message size is exceeded to be a bit confusing as well. Can a new return code be created for this situation (instead of using the IOTHUB_CLIENT_CONFIRMATION_MESSAGE_TIMEOUT return code)? Or even better can the message size be validated before being sent by the IoTHubClient_SendEventAsync method so that you don't have to wait for the callback to get the feedback?

I'm currently using the 2018-06-26 branch of the repo, but I did take a look at the inc for master to check for any updates to iothub_message.h.

Thanks in advance for your help.

[jspaith]

These are straightforward & good questions here and it's odd I'm tying so much below, but there's some background here.

You're correct in that nothing has changed since 6/2018 around this.

On the first question, there's not a reliable mechanism for determining this up front, at iothub_message.h "object" time for caller. At issue is that the message object is just a glorified struct with some helper functions around it. The real magic happens at the protocol layers - amqp, mqtt, http - and this is where they get encoded for appropriate wire protocol.

For an error on max message size before being sent, over AMQP you should (in theory) get an early error (though obviously not cleaner one you propose). Here we check against what the amqp link told us the maximum size is and if a single message won't fit, we error right away. MQTT however has no such mechanism to communicate such a maximum.

For policy configuration areas that we can't get over a protocol, we in general let the server fail rather than us hardcoding constants (e.g. max MQTT message size) and failing. The server can change these values with configuration knobs, whereas the SDK trying to guess its policy risks getting out of sync with the source of truth in Cloud.

The final request about a better error code is fair but a bit trickier than on surface. It'd be possible over AMQP since we know a max, but MQTT is a considerably simpler protocol that doesn't communicate back detailed errors. Are you using MQTT or AMQP?

[zoe-mcelderry]

Thanks @jspaith for your detailed answer.

Whilst I understand your design decisions, this makes life a bit tricky in a deployment scenario where bandwidth could be very limited and the goal is to optimise that by sending fewer, larger messages. If the data has to be sent up (in the case of MQTT) before getting a response that too much data has been sent, it rather defeats the object of maximising the message size to optimise bandwidth.

In terms of which protocol I am using, the intention was to let the end-user decide that, not restrict it to one specific protocol.

It also seems that the throttling system blocks you out if you send too many oversized messages, which makes it hard to establish where the boundaries are by trial and error.

In any case, thanks for your response -- you have answered my questions.

[jspaith]

OK - happy we could help with background if not necessarily the ideal solution.

[az-iot-builder-01]

@zoe-mcelderry, thank you for your contribution to our open-sourced project! Please help us improve by filling out this 2-minute customer satisfaction survey