Two K8s Mimir cluster pointing to same s3 bucket?

grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.

https://grafana.com/oss/mimir/

GNU Affero General Public License v3.0

4.07k stars 519 forks source link

Two K8s Mimir cluster pointing to same s3 bucket? #5243

Closed zakariais closed 10 months ago

zakariais commented 1 year ago

Hey, I want to ask what will happen if two mimir cluster point to same s3 bucket with same storage prefix, will the second mimir import the old data and write new one and will the data be over written? The idea is to launch a new mimir when we upgrade k8s cluster, as we create a new cluster and launch all the resources there instead of existing one. How do you guys think we should handle upgrade and moving mimir cluster from one k8s to another? What will be the behaviour of mimir if two cluster point to same bucket with same storage prefixes?

dimitarvdimitrov commented 1 year ago

Ingesters from the two clusters can upload blocks and those shouldn't conflict as ULID collision is very unlikely.

The only two sources of conflicts that I can think of are

The compactors in how they update the bucket index and delete blocks. If you make sure that there is one set of compactors working against the bucket, then that shouldn't be a problem. i.e. run the compactors only in one cluster.
If the rulers evaluate the same rules you may end up with duplicate samples for the same timestamp. The compactor should choose one of the values value when compacting blocks, but I don't think it's deterministic which one.

@pr00se or @aknuds1, do you have more insights into this, since you did some similar work recently?

Eve832 commented 1 year ago

@osg-grafana this might have docs impact, so tagging you just in case

pracucci commented 1 year ago

I want to ask what will happen if two mimir cluster point to same s3 bucket with same storage prefix, will the second mimir import the old data and write new one and will the data be over written?

This is a setup Mimir was not designed for and we haven't tested. You can give it a try but it's an unsupported setup and I don't recommend it. Dimitar reply above matches what I would have commented as well, but without extensively testing it there may be other things that could break in a subtle way that we can't think of right now.

osg-grafana commented 10 months ago

Per @pracucci and @krajorama, this is an unsupported feature and therefore does not require documentation.

matt-gp commented 7 months ago

Did anyone actually try this???

I have the same situation where I have to do a Blue/Green EKS upgrade.

I'm thinking:

Mirroring the traffic from one cluster to another but leave the ruler/alertmanager/compactor off on the new cluster.
Then after 24hours or so flip the DNS, bring up the ruler/alertmanager/compactor on the new cluster and downscale ruler/alertmanager/compactor on the old cluster.

I understand this isn't a recommended approach but I don't really have another option as I can't do an in place EKS upgrade (long story) so any advice would be extremely appreciated.

Ca-moes commented 3 months ago

Got the answer from here: https://github.com/grafana/mimir/discussions/6144

Attempted to connect a different Mimir instance to the same S3 bucket and was not able to query old data. After waiting around half a day it went all back to normal.