Create a Local User Interface to Display Basic Sensor Status and Error Information

[keenanjohnson]

After lots of excellent learnings from the Frog Sensor Alpha Test Users, it is obvious that we need some additional error/status information to be displayed for each sensor to debug why it might or might not be working.

I'm thinking this should be a local web server that displays some basic information to help diagnose the following issues:

• Sensor Connection Issues - Is there a good connection to the CO2 Sensor, Barometer, and GPS?
  • The simplest version of this would probably be to just display the container std output so we can view the error messages
  • A more complex version would be a web UI that polls a status service for each of the sensors.
• GPS Location Fix - Does that GPS have enough satellites for a good location fix?

Additionally, it would probably be useful to display some other information like:

• Local IP address
• Software Version
• Wifi Connection
• Signal Strength
• Software update status

[keenanjohnson]

I created a thread in the Balena Forums to see if there is a good starting place to build this application from: https://forums.balena.io/t/good-starting-place-for-local-balena-device-ui/349763

[Rabscuttler]

LED status indicator I agree with all the above, but a thought did cross my mind that the classic first indicator IoT devices tend to have is some kind of status LED. E.g. solid green: all is good, amber: pairing, flashing amber: something else, red: broken. This could be more accessible, possibly easier, but would give less information.

Local data for Frog It could be a nice feature to show the device data on that same local server (even if it is just queried from central db) to give a sense of a 'local' dashboard

Accessing webserver How does the user know how to access the local webserver? My router (fritz.box) is able to point a domain to its admin IP (you write fritz.box in a web browser) as it controls DNS - see a lengthy discussion here on the feasibility of this for devices with a few options.

[daveb377]

Blinky LEDs are always helpful!

[keenanjohnson]

Indeed. There are already some status leds on the raspberry pi itself but we probably could use some brighter ones that are visible from the outside of the enclosure for some status. I’ll create a separate issue for that.

On Mon, Oct 18, 2021 at 5:23 PM David Bengtson @.***> wrote:

Blinky LEDs are always helpful!

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Ribbit-Network/ribbit-frog-micropython/issues/7, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATQ3FUABWZTSV43WQHWXCTUHTQETANCNFSM5GHPDLTA .

[djgood]

Accessing webserver How does the user know how to access the local webserver? My router (fritz.box) is able to point a domain to its admin IP (you write fritz.box in a web browser) as it controls DNS - see a lengthy discussion here on the feasibility of this for devices with a few options.

I really like mDNS (mentioned in that discussion) but native OS support does seem iffy. A suggestion I have is we could have is a really simple utility app that has its own mDNS implementation independent of the OS (there's a few libraries out there). That way we can advertise mDNS from a container on Balena and can easily discover it from the app. The app can query a few APIs running in that container to determine if various components are up and running.

Not as easy as hosting a web server on the device but maybe more convenient for users?

[daveb377]

How about a small OLED display for this type of info? Avoids all the configuration issues?

[keenanjohnson]

A small display could work well, though it definitely increases the power budget when we start thinking about a solar-powered version of the sensor, so I'd like to try and find something that could work well for both: https://github.com/Ribbit-Network/ribbit-network-frog-sensor/projects/2

[keenanjohnson]

I created a new issue Ribbit-Network/ribbit-network-frog-sensor#59 for adding a better status LED to the Frog Sensor.

[keenanjohnson]

From the balena forum post, the following helpful links have been added:

Here are some starting points for using a web page to display information about your device:

• balenaDash: balenaHub: an easier way to find and publish fleets and projects for edge devices
• The xserver block: GitHub - balenablocks/xserver: A simple X11 server block
• Setting up a simple NodeJS webserver on a Raspberry Pi 3: Get started with Raspberry Pi 3 and Node.js - Balena Documentation

[keenanjohnson]

Actually, it doesn't really seem like any of the links above are useful.

[keenanjohnson]

This seems to be a useful project to look at for some inspiration: Tools folder creates a local web server: https://github.com/just4give/birdwatcher/tree/master/tools

We can set the local DNS hostname like this: https://github.com/just4give/birdwatcher/blob/f83a4483cfd1a0a5e8ecf70f57a27e1be003d763/docker-compose.yml#L57

[abesto]

(sorry for wall of text)

Here's a web infra person's perspective: from reading this thread, I think what you want is (the beginnings of) a monitoring solution for individual Ribbit sensors. Sure, exposing current state is good, but you'll soon want to see historical trends: when did it break? oh, we lose connectivity every 24 hours. Or every time there's a storm. Hey wait, every time we skip submitting data, CPU usage seems to be maxed out.

You can only make these observations by collecting historical data. You'll also very soon want alerting: once a sensor is up and running, you won't be walking out to it several times a day, or even checking its status page in a browser. You'll want something to let you know when something goes wrong.

Instead of starting to build from scratch and paying the increasing costs of an in-house monitoring system, I'd recommend considering a look at existing open-source solutions. This doesn't mean that everything needs to be configured in one go, but I do strongly suspect that over the lifetime of Ribbit Network this will be much cheaper (in terms of engineering work). I have exactly zero experience with monitoring small hardware, so grain of salt, but here's what a quick search tells me:

• The big boy on the OSS infrastructure monitoring block nowadays AFAICT is Prometheus (certainly in the web / distributed systems space). There are a bunch of articles out there (like this one) about using Prometheus + Raspberry Pi.
• https://elinux.org/RPi-Monitor seems to be a monitoring system built specifically for RPis.

Random thought this triggers, by the way: the Ribbit network itself actually seems like it would fit the architecture of a monitoring system (like Prometheus) pretty well. Collecting data from many hosts, shipping it to a central location, displaying and aggregating and filtering that data is what these tools do. I don't know that this is a good idea, I'll leave that up to you, but it's certainly feasible:

• each Ribbit sensor could expose (using Prometheus as an example) a few Prometheus exporters with data from its sensors, and possibly operational metrics
• a central Prometheus collector would pull data from each sensor. A public Grafana site can then display that data on a map and allow easily creating all kinds of visualizations.
  • there would be a need for some kind of registry / discovery system; this is not particularly hard, but needs to be mentioned, since Prometheus likes to pull data instead of pushing it
• the central Prometheus collector could in theory also collect operational metrics and show them through the same Grafana (though maybe this is less useful as losing internet connectivity from the sensor would mean we also lose operational metrics)
• the Ribbit project could also provide an easy-to-configure Prometheus collector / Grafana dashboard that would live on the local network of the sensor for better monitoring. This could then be configured by the owner of the sensor for alerting (and the Ribbit project could provide examples / collate best practices). Depending on the performance overhead, all of this might even live on the same device; or maybe the sensor just provides an exporter for operational metrics, and the rest of the setup lives elsewhere on the network. The same data could drive any kind of display (leds, on-device OLED, ...)

This would allow leaning on years of experience and work put together by engineers using these tools to monitor all kinds of systems, instead of having to think about the "collect and use time-series data" domain (things like do we buffer data when the network connection drops out intermittently? How do we actually build the right kinds of visualizations? Oh now we need to implement four different notification methods and a way to configure them). It does bring in other kind of work (i.e. configuring and using the existing monitoring systems), but also: that would be much easier to find people to help with, as they're well-known and well-documented solutions used in the industry.

[nucleardreamer]

Hey all, I wanted to update on this issue about a project we are pushing on currently, a heavily related device ui. We are starting from the device-info/configuration route right now, but the goal is for this project to be a good landing place to visualize all the things ribbit needs to provide a nice user experience in a locally hosted page. I would love if anyone felt like they wanted to get involved to push some priorities in terms of this project; that's the goal ultimately!

We are still in the building-ish phase, but device-info, wifi configuration, etc. is mostly working or in the works. We are tracking idea/issues here as well.

[abesto]

Did a little research. It seems that the pieces to build a great monitoring solution kinda sorta exist:

• https://github.com/balenalabs-incubator/balena-prometheus-grafana is an on-device self-monitoring solution using Prometheus and Grafana. The repo is about 3 years stale at this point, but should still be at least an OK starting point.
  • Metrics exposed on a webserver running on the device itself
  • Somewhat configurable. We'd probably need to bolt on some extra pieces of functionality to allow device owners to easily define their own notification sinks (i.e. "please email the owner of the device if there's a warn or higher-level alert")
  • Can be extended to expose whatever metrics we deem interesting
  • Potentially parts of the current custom InfluxDB measurements reporter could be replaced with Prometheus, to reduce the number of moving components in the overall system.
• https://github.com/balena-io-examples/baletheus implements a Prometheus service discovery solution for Balena Cloud. Also fairly stale, but should also be an OK starting point. At a glance it'd need some updates and tweaking to suit us, but the core of the problem seems to be solved.
  • This means we could deploy a version of this to a privileged Balena device (or preferably a proper server), and aggregate data from the Prometheus instances of each individual device for centralized monitoring & alerting ("please email the Ribbit Network oncall if more than 50% of all devices are offline")
  • This would also be the place to implement infrastructure monitoring beyond the Balena devices (i.e. the Ribbit website, data backends, etc.)

Hierarchical Prometheus setups are not a novelty. This approach solves both monitoring-for-the-device-owner and fleet-wide monitoring with a single, battle-tested software stack (plus some glue code).

Actually in hindsight this looks like such an obviously good idea that the fact I'm failing to find any examples / documentation / plug-and-play solutions for it makes me question whether I'm missing something fundamental that would cause it to not work. What are other peoples thoughts on this? (For context: I have minor-moderate experience with Prometheus and Grafana, minor experience with Balena Cloud, and extremely minor experience with smol computers).

Edit: maybe this is a good time to check in with the wider Balena / Balena Cloud community to see how other open fleets deal with this?

[keenanjohnson]

FYI this repo is making good progress towards a device UI that can easily be deployed via Balena. This seems like the most promising starting place to me:

https://github.com/maggie0002/balena-device-ui

[keenanjohnson]

@damz and @akhilgupta1093 I plan on starting with this soon. THe first major decision I believe is that we have to select a transport protocol to move the data from the python script here into the browser client.

In Balena world, I would have created a local MQTT client and then published the data into the MQTT bus, which could then be consumed by things like the golioth client, or a local logger, or the local web UI. I was basing that design pattern off of the balena sense project here.

However, MQTT might not be possible in the micropython environment as it's not really designed for the resource-constrained area, so we'll need to pick another transport protocol to move data between the web front-end client and the main python script.

Any opinions on what would be best?

Options I thought of, though I'm open to others or ideas I've missed:

• I see that there are examples in the micodot repo (our web server) for using websockets, so perhaps that would be the best option for keeping the data live updating?
• We could just set up normal HTTP GET calls as well using the microdot framework for our sensor data? This would be the least complex thing technically, though would probably result in the most code written to handle all the requests.