canonical / grafana-agent-operator

This charmed operator automates the operational procedures of running Grafana Agent, an open-soruce telemetry collector.
https://charmhub.io/grafana-agent
Apache License 2.0
4 stars 8 forks source link

Add physical disk removal alert #134

Open gabrielcocenza opened 2 weeks ago

gabrielcocenza commented 2 weeks ago

Issue

In a case that a disk breaks and is not recognized anymore by the kernel, the metrics of a specific device will stop from been collected.

Solution

Create a Prometheus alert rule to operators be able to notice that a physical disk was removed.

Context

Virtual disks don't have the model, serial and wwn, so those fields are used to just target physical ones. If a physical disk is removed, the alert will continue to fire as long as the disk remains missing and up to 30 days from the removal.