Consider Grafana for System Metrics

[RobHooper]

Grafana is a web interface for visualising data, perfect for our Prometheus system metrics.

Grafana will make it much easier to use and interpret server monitoring, including:

• A time picker updating all graphs at once
• A server picker to filter all graphs to one server at a time
• Simple URL, the queries are stored server-side so we will no longer need long complex custom links.
• A simpler interface will make it easier to use in an emergency.

To do:

• [ ] Deploy Grafana
• [ ] Configure user accounts
• [ ] Integrate with Prometheus data source
• [ ] Create custom dashboard

We already have custom queries written for the current monitor setup, we can re-use these in the new dashboard.

The Prometheus server will be best to host this assuming it has spare system resources.

[RobHooper]

We now have Netdata installed on each server as part of Dogsbody's monitoring stack. Netdata could replace Prometheus and the above proposed Grafana install - needs investigating.

[jpmckinney]

Good reminder. Yes, it sounds like it will be simpler to switch to netdata than to setup (and maintain) Grafana.

[jpmckinney]

Noting that if RabbitMQ's management interface metrics are removed, then we would need Prometheus/Grafana. https://www.rabbitmq.com/docs/prometheus