[Optional Spike: 2hrs] Investigate prometheus monitoring and grafana alerting for EBS volumes approaching full - Githubissues

2i2c-org / infrastructure

Infrastructure for configuring and deploying our community JupyterHubs.

https://infrastructure.2i2c.org

BSD 3-Clause "New" or "Revised" License

108 stars 65 forks source link

[Optional Spike: 2hrs] Investigate prometheus monitoring and grafana alerting for EBS volumes approaching full #5006

Closed sgibson91 closed 2 weeks ago

sgibson91 commented 1 month ago

[!warning] To be pursued if https://github.com/2i2c-org/infrastructure/issues/5005 fails

Tasks

[x] Investigate prometheus metrics to track the size of EBS volumes, if deployed
[x] Investigate grafana alerting capabilities
[x] Report on learnings (can be a comment on this issue)
[x] Scope and refine future tasks in the associated epic

Definition of Done

[x] We understand what is required to provide alerting on EBS volume size from a prometheus and grafana perspective
[x] Tasks in https://github.com/2i2c-org/infrastructure/issues/5004 have been refined

sunu commented 2 weeks ago

Yet to test this end-to-end, but I think we should be able to reuse the Prometheus metric exporter set up defined in https://github.com/2i2c-org/infrastructure/blob/main/helm-charts/basehub/templates/home-space-reporter.yaml by mounting the whole NFS disk as a volume mount.

sgibson91 commented 2 weeks ago

Next steps defined in https://github.com/2i2c-org/infrastructure/issues/5062