MQTT overhead and limit in message size

[tuloski]

Board: MKR1400

I'm using MQTT protocol (With PubSubClient library). I noticed two problems:

• The overhead per single publish is very big. Publishing a payload of 10 bytes takes 250ms (blocking code), while publishing a payload of 500B takes 300-350ms. This kills completely the role of MQTT which should be a lightweight protocol to send lots of small data, for example sensor data. With this overhead it's impossible to send more than 3-4 messages per second and with the blocking code most of CPU time is lost in the delay. Do you think there is a workaround to reduce that time per publish? Where do you think it comes from?
• Limit in payload size: it seems the client at some point blocks the "write" if the dimension is bigger than 500 bytes (I don't have the precise number, I know that 500 works, but at 700+ it doesn't). It's not a limitation in PubSubClient. I wasn't able to track down to the client library where the write is failing.

[SRGDamia1]

PubSubClient sets the maximum outgoing packet size at 256 bytes. See lines 24-27 of PubSubClient.h. You can change that by defining MQTT_MAX_PACKET_SIZE yourself or calling setBufferSize(). That applies to any call to one of the publish(...) functions.

Since you mentioned sending out 500 bytes at once, I'm assuming you're using startPublish/write/endPublish instead. When you're using the write function with TinyGSM, you have to be careful NOT to use write("...") you should use client.write(buf, len) or even client.write(String("...")). (PubSubClient does this correctly using the standard publish(..) command.) If you leave the write content in a character array instead of grouping it into a buffer or even a String, then every character will be sent out individually, typewriter style. For u-blox modules before every single outgoing chunk of data there is a mandatory 50ms wait between when the library tells the module it want's to send data and when the module provides a prompt for the data. Since this library makes not attempt to multi-thread or be non-blocking, if you're using a write("...") command, that's right away 50ms down the tubes for every ... single ... character .... Apart from taking a long time, it also consumes more of your paid data and some servers get testy about receiving the data so far apart and will reject it, especially as the time stretches with more characters.

There isn't a limit in this library for how many characters you can send in a single call to client.write(buf, len). The u-blox module itself limits you 1024 bytes. (See the description of the "Write socket data +USOWR" command in the AT manual if you're interested. It around 25.10.4.) Relative to the total memory of even a SAMD21, that's a fairly big single buffer or String.

If you're already sending things out in a single buffer or String and you're still getting the same really slow sends and limits, let me know and I can dig further.

I hope this helps.

[tuloski]

I'm sending it out as a single String. I'm using mqtt.setBufferSize(N), tried with N= 1024/2048/65536. I was in fact able to send out 500B without the limitation of 256 Bytes. BTW thanks for the 50ms explanation, I found it on the manual and (small fix) it's at section 26.10.4, not 25.10.4 (also in the comments in the library points at section 25 instead of 26). Maybe they updated the manual recently. So can it be the 1024 bytes limit of u-blox? Or if bigger it should send it as multiple data (so adding 50ms each block of 1024)?

About the overall 250ms "delay", do you thinks it's ok, or can I try to optimize something here and there? Because with messages @2 Hz it's 50% of the cpu time blocked, and I risk to lose I2C and serial data meanwhile.

[SRGDamia1]

The 1024 is a hard limit on the ublox. If you try something bigger, it will fail. You'll need to use start/write/end publish with multiple writes less than 1024.

Interrupts are still active, so you shouldn't lose serial or I2C data unless your other buffers overflow.