search

observatorium / thanos-receive-controller

Kubernetes controller to automatically configure Thanos receive hashrings
Apache License 2.0
93 stars 42 forks source link

Support annotating pods on change for thanos receiver router and ingestor setup #125

Open rtnpro opened 7 months ago

rtnpro commented 7 months ago

Currently, thanos-receive-controller uses the same label to watch receiver pods and update them on hashring changes.

However, in a Thanos receiver router and ingestor setup (https://thanos.io/tip/proposals-accepted/202012-receive-split.md/), while we need to watch the ingestor pods to update the hashring configmap, the hashring configmap gets used in the router pods. In this scenario, the router pods need to be annotated on hashring change instead of the ingestor pods.

For thanos-receive-controller to support this Thanos receiver router+receiver setup, it needs to support an option to specify a separate pod label (different from the pod label used to watch the receiver ingestor pods) to use for annotating pods on hashring change.

rtnpro commented 7 months ago

Here's a draft PR: https://github.com/observatorium/thanos-receive-controller/pull/126

rtnpro commented 6 months ago

@matej-g I need your thoughts on this issue.

We recently implemented the proposal to setup Thanos Receiver as Router and Ingestor for making Thanos Receiver highly available.

Currently, we are using reloader controller to rotate the receiver router pods on hashring change. It works, but has it's delay.

We have tested the changes in #126 and it works faster and updates the hashring config in the router pods without rotating them.

It will be great if this changes could be implemented in upstream.

Thanks.

miguel-cardoso-mindera commented 1 month ago

I'm also interested in this, has there been any news?

@rtnpro appreciate your good work!

rtnpro commented 1 month ago

I am still waiting on a response and review from the upstream.