Persitence Service Addon (DB based buffering)

[marco2901]

Some Devices are running in very network weak regions or even have only a cyclic transmission rate (e.g. once per day). For those devices it would be good to have an persistence service, which is storing data (not just in-memory) over a longer period and flush them at connection time.

It would be cool to have addons for connecting databases like influx, mongo or round-robin mechanisms based ones, which then could take care about data persistence during disconnected operation time.

[phcrb]

Hi @marco2901, @didier-wenzek, I'm planning to deploy some gateways with thin-edge in places with no wired connection available, so transmission will be performed by a 3G/4G cellular link. The devices attached to the gateway typically send measurements every 5 to 15 minutes. A mecanism to store data locally when the cellular link is not available would be strongly apreciated.

My gateway does not run a debian-like os, but rather a custom 32 bit linux with a simple busybox and no apt facility, built by the hardware manufacturer, so it's nearly impossible to install a complex database engine such as mongo or influxdb as they may need many dependencies. A simple file-based storage buffer would be more adapted to my use case.

[didier-wenzek]

A first step is to configure mosquitto to enable persistence.

For instance, to store on disk all the pending messages every 60 seconds, /etc/mosquitto/mosquitto.conf can be updated with:

persistence true
persistence_location /var/lib/mosquitto/
autosave_interval 60
autosave_on_changes false

If you have few but important messages, you can persist the pending messages more frequently or when x new messages are pending. Say every 5 messages:

persistence true
persistence_location /var/lib/mosquitto/
autosave_interval 5
autosave_on_changes true

For a more robust solution, using a local small foot print database, one also needs some kind of application acknowledgement from the cloud peer telling the device when these messages have been successfully transmitted and can be safely removed from the local database.