search

observatorium / thanos-receive-controller

Kubernetes controller to automatically configure Thanos receive hashrings
Apache License 2.0
93 stars 42 forks source link

Add annotate pods on hashring change feature #92

Closed matej-g closed 1 year ago

matej-g commented 1 year ago

Signed-off-by: Matej Gera matejgera@gmail.com

This PR adds a feature to annotate the receiver stateful set pods on a hashring update. It adds an annotation with latest hashring configuration hash.

The point of this feature is to (optionally) enable faster configmap syncs. By default, although configmap is updated instantly via API, the mounted volumes take some time to reflect the change (according to the documentation this can be up to 1 minute kubelet sync interval + 1 minute TTL cache: https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically). However, having discrepancies for longer periods between the replicas can cause increased error rates during scaling up / down. The point of this change is to minimize the window during which the configmap can be updated. Since internally, the receivers can be configured to watch for configmap changes on local volume with shorter interval, this feature should realistically bring down the delay until the hashring is updated ~= config watch interval (whereas now it would be hashring update <= kubelet sync + cache).

Alternatives considered More proper way of doing this would be changing the kubelet sync and / or caching configuration, but the issue is 1) I'm not able to do this in all my environments; 2) It could have other adverse performance effects on the system

Some related conversation on configmap updates and annotations: https://github.com/kubernetes/kubernetes/issues/30189

moadz commented 1 year ago

Confirmatory evidence that this works

In the below image we can see your change tested against master (with some modifications to always update config-map on change, --receive.hashrings-file-refresh-interval=5s

We deleted the pod to see how long it would take for the individual receivers to pickup the config change. The expectation with the deletion is that you will have a config hash = all receivers - thanos-receive-3 and then the configmap should return to the original value when thanos-receive-3 is back online.

What we see on the left is this behaviour. On the right we can see that some of the receivers never pickup the change all-together.

image