When a large WAL is already present, prometheus can take some time before actually be ready because it replays its WAL.
BUT: prometheus-elector claims leadership as soon as it starts. So if Prometheus takes some time to start, this can prevents falling back to another replica, which is already ready.
What could be nice is to wait for the sidecar Prometheus to be ready before actually trying to claim the leadership.
jlevesy
commented
1 year ago
What's wrong
When a large WAL is already present, prometheus can take some time before actually be ready because it replays its WAL.
BUT: prometheus-elector claims leadership as soon as it starts. So if Prometheus takes some time to start, this can prevents falling back to another replica, which is already ready.
What could be nice is to wait for the sidecar Prometheus to be ready before actually trying to claim the leadership.