QuentinBisson
commented
2 months ago To unblock Mimir:
- Release announcement for mimir changes and opsctl open removal
- Adding missing datasource to all dashboards and finish atlas alert reviews
- Clean up the heartbeats per cluster and clean up prometheus heartbeat alert and remove the heartbeat creation
- Add information about the grafana agent in the heartbeat ops-recipe
- Prometheus to grafana cloud prometheus needs an alert and an ops-recipe update
- Explain in v20 release note that the prometheus agent resource usage will grow bigger
- Finish removing the prometheus on the MC to reduce resource usage
- Add HPAs on components we can but the ingester and add an ops-recipe on how to scale it up and down and also that we are alerted when we need to scale it up or down
- Review opsrecipes around prometheus/mimir
- Try out Mimir on Goat (private MCs)
- Remove the mimir specific opsgenie policies
Rotfuks
The goal here is to replace the current monitoring on a testing CAPA installation and have Mimir in place to send alerts and visualize data via Grafana.
This solution should be fully automated, so we can replicate this setup on multiple installations by using a toggle flag which would trigger the deployment and configuration of this monitoring solution. Automation for the new monitoring setup should no be added into the existing prometheus-meta-operator.
Here are some goals to reach :
After Mimir is succesfully setup on one installation, we can start getting our internal customers onboarded and check the new system before rolling it out everywhere and cleaning up the old system.