Closed gtato closed 1 month ago
Steps to reproduce on GKE:
juju ssh --container postgresql postgresql-k8s/leader bash # leader
apt update && apt install nano curl -y
nano /var/lib/postgresql/data/patroni.yml
# Remove the other units from both pg_hba sections.
curl -X POST localhost:8008/reload # leader
# Wait 30 seconds.
juju ssh --container postgresql postgresql-k8s/0 bash # replica
apt update && apt install curl -y
curl -X POST localhost:8008/reinitialize
juju ssh --container postgresql postgresql-k8s/1 bash # replica
apt update && apt install curl -y
curl -X POST localhost:8008/reinitialize
The issue is related to the permissions in the volume mounted in the units, like in https://warthogs.atlassian.net/browse/DPE-707. I'll create a PR to fix that.
Hi, @gtato!
Revisions 332 and 333 from the 14/edge
channel contain the fix for this issue.
Steps to reproduce
This happened in prod, and I haven't reproduced in a local env.
At some point replicas go out of sync and try to restore pgdata from the primary, but fail with this error:
Expected behavior
Replicas retrieve correctly the wal entries from the primary and restore their state.
Actual behavior
Replicas fail to get pgdata and fail. This in turn causes the primary wal to increase the size until it can't function properly.
Versions
Operating system:
Juju CLI: 2.9.49
Juju agent: 3.1.8
Charm revision: 14/edge 198
microk8s: v1.26.15
Log output
Juju debug log:
Additional context
To resolve this issue I used these steps: https://matrix.to/#/!BukWfnyOTgQSKAxdtT:ubuntu.com/$C-iLZEXS39xBD8vFV40EVBWFefNjlvUQmxFDxNcS2p0?via=ubuntu.com&via=matrix.org
but the I am not sure how to prevent this issue in the future.