Open Rotfuks opened 3 months ago
I think it would be nice to run mimir continous testing (similar to loki canary) for e2e tests
Yeah that's a nice idea :)
I am running mimir continuous_test on grizzly with the following config:
mimir:
continuous_test:
enabled: true
auth:
tenant: anonymous
and this renders the following metrics:
and we could just use those alerts https://github.com/grafana/mimir/blob/f52911d917c8c52e0da6a59348a64dd7f7622072/operations/mimir-mixin-compiled/alerts.yaml#L1097
The only downside is that we need to wait for the next minor helm chart release or use a weekly version because this https://github.com/grafana/mimir/pull/8654 is not yet released
So what's the best plan of action IMO is to wait for the continuous testing to be a default config for our mimir before doing anything else. In the meantime, I'll create a dashboard using the metrics from the rules' mixins and if it's good enough, I'll think about pushing it upstream as a mixins dashboard.
I think so yes, maybe we can have a pr ready with the alerts? The mixins contains some that could be useful
@QuantumEnigmaa we decided in retro to use the chart version rc0 for now but keep the old image of mimir 2.13
All good with me 👍
We can start this again once we're done with multi-tenancy :)
Motivation
n order to raise our confidence in the stability of our observability platform and be sure that our ongoing work and releases won't negatively impact our observability platform operations we need to create extensive tests giving us early feedback loops. As mimir is one of our core components, we should make sure it's thoroughly tested.
Todo
[ ] Investigate some good initial test cases to give us feedback on the stability of a release on mimir on CI. This can be:
[ ] Implement those CI test cases
[ ] Investigate some good initial test cases to give us feedback on the stability of a release before releasing. This can be:
[ ] Implement those test cases
Make sure to stay with a set of minimal but valuable test cases, nothing to detailed and fancy.
Outcome