☂️ Use the prometheus-operator to manage Gardener's monitoring stacks

[wyb1]

Stories

• As developer I want better means to define what to scrape and what to visualise (better than hard-wired).

Motivation

Switching to the Prometheus operator promises a huge improvement for how components can define what to scrape and what to visualise.

Acceptance Criteria

• [x] Investigation

  Tasks

  - [x] https://github.com/gardener/monitoring/issues/16

• [ ] Documentation

  Tasks

  - [x] Create a [GEP](https://github.com/gardener/gardener/tree/master/docs/proposals) for this in Gardener https://github.com/gardener/gardener/pull/6151

• [ ] Implementation

  Tasks

  - [x] https://github.com/gardener/gardener/issues/6308 - [x] https://github.com/gardener/gardener/issues/6318 - [x] https://github.com/gardener/gardener/issues/6319 - [ ] Migrate each component to BYOMC - [ ] Seed Monitoring

Enhancement/Implementation Proposal

• https://github.com/gardener/gardener/pull/6151

[timebertt]

General remark/advice: when implementing the GEP, you probably want to consider upfront how the feature will be verified in e2e tests. E.g., you could add some (dummy) ServiceMonitors in provider-local (similar to how other extensions would do it) and add some verifications to the default e2e, that talk to prometheus and ensure the required metrics are available. This is similar to what we do for testing the CA rotation feature parts that are contained in the extensions library :)

This will greatly improve the stability of the feature right from the beginning. Considering that this feature will probably not be tested manually during development, it is easy to regress. So having test coverage for that will significantly ease our lives :D