search

RubyDevInc / paho.mqtt.ruby

Eclipse Public License 1.0
31 stars 19 forks source link

Offline Buffering? #24

Open rgaufman opened 6 years ago

rgaufman commented 6 years ago

I found on this mailing list post that paho supports offline buffering: https://dev.eclipse.org/mhonarc/lists/paho-dev/msg03914.html

Where can I find more information about this? - what local store is used for the offline buffering?

p-goudet commented 6 years ago

@rgaufman The offline buffering features enables to buffer message or acknowledgement to be temporary store in case of network failure. Once the connection is reset, the buffer packet would be send. The client should be configure in persistent mode (persistent: true) with clean session off ( clean_session: false). The packet are buffered in a local queues.

rgaufman commented 6 years ago

Is there any way to persist this data to some persistent store? - e.g. a local sqlite database or something.

p-goudet commented 6 years ago

I understand the need for an application to (temporary) store data in an offline state. However I don't think that it should be include in the MQTT client. The offline buffering in Paho aims to store packets that should have been sent when a network failure occurs. If the network failure keeps going or could not be fix in a short time, the application which use Paho should notice the network failure. Then, the application could localy store data during the offline time in a database such as sqlite. I think that the application should stop create (and store) MQTT packet if the MQTT broker could not be reached.

Could you tell me more about why do you need a persistence local storage?

rgaufman commented 6 years ago

This page explains it: https://www.hivemq.com/blog/are-your-mqtt-applications-resilient-enough/ - especially the section "Offline Buffering".

But basically, the functionality of QOS1 and QOS2 can be lost if the MQTT library can't gracefully handle connection loss and cache the messages in an offline persistent cache. The concept of persistence is also lost as the message sent is not really persistent, it will be lost if the process dies or the unit is rebooted while offline.

So if paho can't send a message for any reason, can I add some kind of callback to manually store this message in an offline cache? - can I also add a callback when the connection is restored to then re-send all those messages from the offline cache?

p-goudet commented 6 years ago

Thank you very for this docs. In order to separate the business logic form the transport layer, the MQTT client might have to store data in local back-end in case of offline state. This would add extra dependencies (sqlite, ORM....), however it could provide are better offline handling. Currently, in persistent mode, the client would try to automatically reconnect, if the connection is re-established, the message and acknowledgment queued during the offline time are resend. However it could be improved by a adding a separate local storage used when offline mode is noticed by the MQTT client. I think that such a feature should be add in near feature.

sudamerushabh commented 4 years ago

Have anybody implemented full proof solution for the above case for offline buffering using sqlite or any other database?? There are some projects like Fluentd,etc which provides offline on disk persistence solutions. But they are not that reliable as well.

Any help would be appretiated.