search

DoESLiverpool / somebody-should

A place to document practices on the wiki and collect issues/suggestions/to-do items for the physical space at DoES Liverpool
32 stars 11 forks source link

Provision a reproducible MQTT broker Rpi that can be used off-site #1110

Open goatchurchprime opened 5 years ago

goatchurchprime commented 5 years ago

Rather than reopen the Issue #616 we can keep progress updated here as to where we want to go.

@ajlennon is building some kind of ballina thing that can rebuild and deploy to the rPI that is hanging on the front wall with a 7" touch screen.

Some details of what it does (not how to rebuild it) are at: https://github.com/DoESLiverpool/somebody-should/wiki/MQTT-services This includes running a DoESLiverpool-MQTT wifi, which makes the thing portable.

We've decided not to host the node-red or database instances on this rPI because we don't want failures with those to bring down the core MQTT-broker service anymore. These are going to be put on some other computer or other screen.

Now, we've got to show something useful on the screen. If the whole GUI is not going to be there, then it's either got to be a constantly scrolling mosquitto_sub -t '#' dump, or we can run this little script that is uses curses to plot a constantly updated view of the different messages going through.

https://github.com/goatchurchprime/jupyter_micropython_developer_notebooks/blob/master/projects/mqttmonitortext.py image

ajlennon commented 5 years ago

OK so I've been making progress with this with @goatchurchprime ... Some status updates.

We have been taking down the 7in dashboard display on the wall to take to various "show and tells". Also we have been updating this and in the process we've managed to break it requiring reinstalls.

We would like to be in a position where we have the data from the space persisted in a reasonably reliable fashion so that if (when) things break or we modify the system we don't lose the data and we don't stop storing data.

As such we've broken the system out into two different pieces of hardware (for now - we might break things out further in future depending on how the system performs).

There are a couple of niggles to be fixed - on first boot the browser comes up before network connectivity is established. We probably want to hold off bringing up the browser until this happens. Similarly if connectivity is lost when Grafana dashboards are being displayed they seem to stop updating and we need a way to retry in this event.

The Balena/Docker configuration for this is currently here.

https://github.com/DynamicDevices/does-rpi3-dash

In theory this can be deployed trivially to give us web-displays to any number of new devices (such as Raspberry Pi's) which are supported by Balena.

Mosquitto - MQtt broker which Sonoff Espurna devices in the space are publishing to on mqtt.local

InfluxDB - NoSQL database we are experimenting with to persist data from Mosquitto

Node-Red - Which we are using to wire up flows to tranform and persist certain message data i.e. bridging from Mosquitto to InfluxDB

Grafana - Used for graphing and data-visualisation of the datapoints in InfluxDB

There's also an example "Bridge" piece of Python scripting to show how we can store MQtt messages to InfluxDB programatically but this is not currently in use.

There are also a couple of niggles with this. Docker containers are non-persistent so when you restart them you revert to the original state and lose any changes (i.e. any data). There is a capability to define a storage mount which is where data to be persisted goes. We have these defined for the containers and so shouldn't lose Node-Red flow definitions, Grafana dashboards or InfluxDB data. I am seeing that when I re-deploy I do seem to lose data though so need to work out if this is happening and if so why.

Also the data is currently persistet to a uSD card in the RPi which is bad practise. Eventually the uSD will fail to to limited flash write cycles. We need to move this to a middle-term solution such as storing to rotating media.

The Rpi is currently uncased on my desk. We should case it up (possibly with a 2.5in laptop HDD) and find a place to put it out of the way.

ajlennon commented 5 years ago

TODO: I also need to update the documentation here

https://github.com/DoESLiverpool/somebody-should/wiki/MQTT-services

DoESsean commented 5 years ago

Seeing as I know literally nothing about MQTT and all that kind of thing, I'll rely on @ajlennon and @goatchurchprime to tell me whether this is still ongoing or should be closed.

ajlennon commented 5 years ago

We're still working on it :)

goatchurchprime commented 4 years ago

JR is setting up the mingstack in his home on his always-on computer, so if this issue has an answer, we could turn this into a note in the https://github.com/DoESLiverpool/somebody-should/wiki/MQTT-services page pointing to the place that has all the necessary provisioning.