[EPIC] - Greenhouse Monitoring & Alerting

[IvoGoman]

Priority

High

Description

Greenhouse controllers emit the common set of metrics exposed with controller-runtime. Additional instrumentation should give more insights into specific error conditions of a controller's reconciliation. Furthermore, there are more components of Greenhouse which need to be actively monitoring such as cors-proxy, id-proxy, and service-proxy.

Metrics should be visualised with Plutono Dashboards so that the overall platform, as well individual components health, is easily consumable.

Metrics should be used to define PrometheusAlertRules so that failure conditions can be identified and proactively resolved.

Acceptance criteria:

• [ ] Instrumentation of the Controllers #283
• [ ] Instrumentation of Greenhouse Components #286
• [ ] Prometheus alerts are defined and can be routed to slack #280
• [ ] Plutono Dashboards are defined #285

Reference Issues

No response

[auhlig]

Yes, please! We should invest at least in the described basic monitoring before onboarding more customers

[auhlig]

• dex controller for authentication related insight