It's currently hardcoded as 1 minute, but pods in my cluster can take longer than that to start up (e.g. when spinning up a new node), resulting in this warning: level=warn caller=main.go:573 ts=2024-01-11T09:24:33.383554728Z msg="failed polling until pod is ready" pod=thanos-receive-3 duration=1m0.010513876s err="timed out waiting for the condition". This results in the Hashring ConfigMap getting updated before all pods are ready.
FYI we have --allow-only-ready-replicas enabled and are on image version main-2023-11-06-c57219e.
Hello!
Is it possible to allow the configuration of the
waitForPodtimeout? https://github.com/observatorium/thanos-receive-controller/blob/c57219e684d688af580e7cb8d7d8e96a8f939c6b/main.go#L605It's currently hardcoded as 1 minute, but pods in my cluster can take longer than that to start up (e.g. when spinning up a new node), resulting in this warning:
level=warn caller=main.go:573 ts=2024-01-11T09:24:33.383554728Z msg="failed polling until pod is ready" pod=thanos-receive-3 duration=1m0.010513876s err="timed out waiting for the condition". This results in the Hashring ConfigMap getting updated before all pods are ready.FYI we have
--allow-only-ready-replicasenabled and are on image versionmain-2023-11-06-c57219e.