FxKu
commented
2 years ago Are both instances healthy? "no switchover candidate found" sounds like no healthy replica has been found.
Closed dobrac closed 9 months ago
FxKu
commented
2 years ago Are both instances healthy? "no switchover candidate found" sounds like no healthy replica has been found.
dobrac
commented
2 years ago I’m not fully sure if the second one is successfully restarted (and healthy) at that moment, but it’s definitely healthy after short time. If I do manual switchover using patronictl, everything is working well.
I haven’t manage to do successful switchover while updating, it fails every time.
bootc
commented
9 months ago I am also running into this issue, but I'm not sure it's exactly like OP. Like @dobrac I can exec into the pod and force a switchover and all is well, but the Operator fails to do the switchover when pods need evicting:
time="2024-01-14T09:34:53Z" level=info msg="starting process to migrate master pod \"gitlab/gitlab-postgres-1\"" cluster-name=gitlab/gitlab-postgres pkg=cluster
time="2024-01-14T09:34:54Z" level=debug msg="making GET http request: http://10.44.95.149:8008/cluster" cluster-name=gitlab/gitlab-postgres pkg=cluster
time="2024-01-14T09:34:54Z" level=error msg="could not move master pod \"gitlab/gitlab-postgres-1\": could not find suitable replica pod as candidate for failover: no switchover candidate found" pkg=controllerThe cluster status from the URI mentioned is:
{
"members": [
{
"name": "gitlab-postgres-0",
"role": "replica",
"state": "streaming",
"api_url": "http://10.44.22.248:8008/patroni",
"host": "10.44.22.248",
"port": 5432,
"timeline": 63,
"lag": 0
},
{
"name": "gitlab-postgres-1",
"role": "leader",
"state": "running",
"api_url": "http://10.44.95.149:8008/patroni",
"host": "10.44.95.149",
"port": 5432,
"timeline": 63
},
{
"name": "gitlab-postgres-2",
"role": "replica",
"state": "streaming",
"api_url": "http://10.44.98.184:8008/patroni",
"host": "10.44.98.184",
"port": 5432,
"timeline": 63,
"lag": 0
}
]
}This is with Postgres Operator 1.10.1 and spilo-15:3.1-p1 on Kubernetes 1.28.5 using Rancher RKE2 on on-prem VMs and bare metal.
It looks like the "state": "streaming" is not accounted for; the code expects "state": "running" to find candidates.
macedigital
commented
9 months ago Thank you @bootc for picking this up and providing a solution :+1:
It seems it might trace down to a change introduced here https://github.com/zalando/patroni/commit/d46ca88e6bb4b4788ec86689026c2d2e4da68c79.
We'll aim to merge the fix next week.
macedigital
commented
9 months ago Thank you @bootc for your contribution!
monotek
commented
5 months ago We still see this problem with operator version v1.11.0.
The "ghcr.io/zalando/spilo-15:3.2-p1" image, where the patrocnictl list output has "streaming" instead of "running" in the state.
Going back to "ghcr.io/zalando/spilo-15:3.0-p1" image sovles the problem
Edit: Nevermind, we updated the operator chart but had set the old operator image explicitly. Works after removing the old image tag and using the charts default image version.
Please, answer some short questions which should help us to understand your problem / question better?
Helllo,
we are encountering an error while cluster updating (any change in the configuration that needs pods restart), where the master is not correctly switched to the synchronous replica. We have 2 instances, one is master and one is a synchronous replica.
Here is log from the operator, the problem occurs every time:
Any idea what could be wrong? It could be definitely some wrong configuration (maybe three instances are the minimum?), but I can't figure out that it is.