Enhance infrastructure with fancy web dashboard

[marmarek]

Add some monitoring to our infrastructure. The infrastructure consists of:

• build servers
• repository master server (yum/deb.qubes-os.org)
• test server: openqa.qubes-os.org
• various random scripts running all over the place (like https://github.com/marmarek/signature-checker, https://github.com/woju/qubes-stats)

Things we want to monitor:

• if the service is running, network connectivity
• disk space
• failure rates (like failed builds)
• freshness of the data (for example detect github hooks deliver issues)
• more in the future

Desired properties:

• passive data collection - monitored host/service/vm should send the data itself, the collecting service should have no direct access to send anything to the monitored service (no extra attack surface)
• consequence of the above: also monitoring freshness of received results
• fancy web dashboard, with easy to read "OK/non-OK" indicator(s)
• alert reporting (at least email)

[marmarek]

@woju any suggestion about the software to use here?

[woju]

Fancy web dashboard is definitely grafana. You need to pick a backend though. I have experience with two backends: Prometheus and graphite/carbon.

Prometheus is very much geared towards pulling: passive collection is at best second-class citizen, maintainers actively discourage it, and the only scenario they think it makes sense is to post the result of batch jobs. Read about their worldview here: https://prometheus.io/docs/practices/pushing/.

Graphite is very simple and indeed push-oriented, you can write the data to it using netcat. It might be what you want.

Third option might be to investigate InfluxDB, which is another supported backend and seems to have external write API: https://docs.influxdata.com/influxdb/v1.6/tools/api/#examples-5 I have no experience with it.

As to the agent, there are multitude of resident daemons which can send metrics to one backend or another. I have used collectd to write to carbon and it was fine. There are others, so it might be that there are better options. For prometheus, the agents (called exporters) are supplied by the project or you can implement a simple HTTP endpoint (probably even ncat/socat+sh would be sufficient).

For the notifications, there are two options. One is to use Grafana, which has reasonable options in itself. Other option is to rely on the backend. Graphite does not have it at all AFAIU. With Prometheus this is important for some HA/distributed databases, which I don't think is the case here.

As to the packaging: graphite-carbon and collectd are simple and generally OK to use from Debian repositories. Grafana and Prometheus have their own apt repositories, which are stable and compatible (I had no issues with dependencies).

https://grafana.com/docs/ https://prometheus.io/docs/ https://graphite.readthedocs.io/en/latest/ https://collectd.org/documentation.shtml

[marmarek]

This is happening!

[rapenne-s]

Is it still an issue? :) It looks like it has been solved already, if you need some help, I'd be happy to give a hand in this area.