search

mlesniew / PicoMQTT

ESP MQTT client and broker library
GNU Lesser General Public License v3.0
255 stars 28 forks source link

SD card support for storing messages at the broker #9

Closed davidkjackson54 closed 1 year ago

davidkjackson54 commented 1 year ago

Just a thought.. Would there be any advantage if using a ESP32 with built-in SD Card - such as the SparkFun Thing Plus - ESP32 WROOM https://www.sparkfun.com/products/20168 for storing messages on that SD Card at the broker instead of storing in memory?

mlesniew commented 1 year ago

Storing messages on a SD card could be useful to implement advanced broker features, especially QoS > 0, retained messages and potentially will messages.

The current broker implementation does not need storage, the messages are not even always loaded to RAM (at least not in full). When a message is published by a client, the broker reads it in small chunks (which are stored in a buffer in RAM). From there they are sent further to the other clients. Once one chunk is fully forwarded, the broker moves on to the next one until the whole message is processed.

This is different from typical brokers, including the popular Mosquitto broker. This behavior is also not really strictly compliant with the MQTT 3.1.1 spec. However, this simplification has several benefits, especially when the broker needs to run on limited hardware. The main advantage is that it can efficiently handle messages of any size with minimal RAM usage.

At the same time, writing all messages to the SD card before forwarding would cause a significant slowdown because of the I/O delays and could cause significant wear on the SD card too.

davidkjackson54 commented 1 year ago

It was just a thought., I personally don't think there is much wear and tear on a SD card,. They are good for quite a few years.
I/O might be a concern but again - it would be an option to incur or not.