How to fix replica not sync in kubegres

[simsicon]

Hi, I found this fatal error in kubegres replica:

FATAL: could not receive data from WAL stream: ERROR: requested WAL segment 00000003000000A400000063 has already been removed

I think I need to run pg_basebackup to reset the cluster, but I am wondering how that action should be performed with kubegres?

Thanks

[simsicon]

By the way, according to this doc https://www.postgresql.org/docs/13/warm-standby.html

If you use streaming replication without file-based continuous archiving, the server might recycle old WAL segments before the standby has received them. If this occurs, the standby will need to be reinitialized from a new base backup. You can avoid this by setting wal_keep_size to a value large enough to ensure that WAL segments are not recycled too early, or by configuring a replication slot for the standby. If you set up a WAL archive that's accessible from the standby, these solutions are not required, since the standby can always use the archive to catch up provided it retains enough segments.

But I can not found the specific config in kubegres, is it essential to avoid the problem I encountered?

[alex-arica]

Thank you for your message.

The setting "wal_keep_size" can be modified from postgres.conf.

Kubegres stores postgres.conf in the ConfigMap: base-kubegres-config.

Once you modified a ConfigMap, Kubernetes does not notify Kubegres about such changes. Therefore, you have to refresh your cluster by following these instructions: https://www.kubegres.io/doc/override-default-configs.html#refresh_kubegres

Note that changes to base-kubegres-config applies to all clusters of Postgres in a same namespace. If you have multiple clusters of Postgres in the same namespace, you can also create your own custom ConfigMap and add a postgres.conf for a specific cluster of Postgres. More details here: https://www.kubegres.io/doc/override-default-configs.html

Few questions which can help me to understand the issue that you are facing:

• How many Pods (instances of Postgres) is your Kubegres YAML configured with?
• Does this issue happen on all other replica Pods?
• Do you see any error in the logs of the Primary Pod?

[simsicon]

Thanks for the your reply, here's the information you asked:

How many Pods (instances of Postgres) is your Kubegres YAML configured with?

I have one kubegres postgresql cluster running, contains 3 pods, 1 primary and 2 replicas

Does this issue happen on all other replica Pods?

Yes, all other replica pods has the same issue and fatal error.

Do you see any error in the logs of the Primary Pod?

I can not fetch and see the origin log of the primary pod from the very beginning of this problem (it had happened for a while, I was on other things and did not try to found out why until now). But the primary pod has a log constantly:

ERROR: requested WAL segment 00000003000000A400000063 has already been removed

but I think this error message is due to replicas are constantly trying to request the WAL segment.

maybe I should just delete two replicas then auto recreate the pods hope it will sync?

[alex-arica]

Thank you for those details.

It seems like the data of the Replica Pods are way behind the Primary Pod. Consequently, Replica is not able to catch-up. This could happen either because a network issue happened for a long period between a Primary and Replica Pods OR if the Primary receives lot of data and is not able to transfer the information fast enough to Replica Pods.

Do you have large data input constantly going to the Primary database?

As you suggested you could modify the size of "wal_keep_size" if you have lot of data input.

There are also some suggestion to increase "wal_keep_segments" in the following StackOverflow page: https://stackoverflow.com/questions/47645487/postgres-streaming-replication-error-requested-wal-segment-has-already-been-rem

Yes, I would delete one replica Pod. Kubegres will create a new Replica Pod automatically. I would check the logs of the new Replica Pod and make sure everything is in sync. If the logs look good you can delete the other failing Replica Pod.

There are other ways with Postgres to replay the WAL data to recover a replica. I suggest to check the Postgres documentation.

This type of issue is very specific to PostgreSql and the way the streaming replication works. I have never experienced it but as you may have noticed there are many reports of it in StackOverflow. Kubegres delegates the replication to PostgreSql so it does not have any feature about recovering a Replica.

[alex-arica]

I made a new change to Kubegres so that it detects this type of issue and replace Replicas with unhealthy replication state. This will work as long as the official Postgres binary pg_isready reports as unhealthy.

To install Kubegres 1.13, please run:

kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.13/kubegres.yaml

[simsicon]

I made a new change to Kubegres so that it detects this type of issue and replace Replicas with unhealthy replication state. This will work as long as the official Postgres binary pg_isready reports as unhealthy.

To install Kubegres 1.13, please run:

kubectl apply -f https://raw.githubusercontent.com/reactive-tech/kubegres/v1.13/kubegres.yaml

Great, Thank you, by the way after applied the new parameter you suggested, I haven't meet the issue yet.