2i2c-org / infrastructure

Infrastructure for configuring and deploying our community JupyterHubs.
https://infrastructure.2i2c.org
BSD 3-Clause "New" or "Revised" License
106 stars 65 forks source link

Enable disk size alerting for AWS EBS volumes on VEDA staging hub #5062

Open sgibson91 opened 1 week ago

sgibson91 commented 1 week ago

Context

Task list

 - [ ] Use [Prometheus metric exporter](https://github.com/2i2c-org/infrastructure/blob/main/helm-charts/basehub/templates/home-space-reporter.yaml) and mount the whole NFS disk as a volume
 - [ ] Create a grafana dashboard to make use of the metrics 
 - [ ] Investigate and enable Grafana alerting based on some threshold, alerts sent to support[at]2i2c.org
 - [ ] Deploy to VEDA staging hub
 - [ ] Test by setting the threshold to something really low and see if an alert is triggered
 - [ ] Document!

Definition of Done

GeorgianaElena commented 1 week ago

@sgibson91, raising here for visibility so you're not caught by surprise in case you haven't seen https://github.com/2i2c-org/infrastructure/pull/4923 (for more context https://2i2c.slack.com/archives/C055A1J1DRP/p1727942694562319)

sgibson91 commented 1 week ago

Thank you @GeorgianaElena I did see it and did remember about it, but hadn't done the work digging it out yet! Thank you!

@sunu, https://github.com/2i2c-org/infrastructure/pull/4923 will need to be reverted for Grafana alerting :)