toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.11k stars 174 forks source link

pgsync process not reducing the transaction logs created by the postgresql setting logical_replication=1 #415

Open smadaan2 opened 1 year ago

smadaan2 commented 1 year ago

I have been using pg_sync for couple of months and have enabled logical_replication to 1 as mentioned in pg_sync documentation. my diskspace is getting filled by AWS RDS postgresql server . Upon investigation I came to know the RDS will retain the transaction logs for external consumers of the logs for replication purpose and if there is no external consumer for these logs, they just remain there forever. We are using pgsync 1.1.32.

Note: Our application data is in MBs.

I tried the following query on my db

select slot_name, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(),restart_lsn)) as replicationSlotLag, active from pg_replication_slots ;

slot_name replicationslotlog. active database1_index1 30 GB false database1_index2 30 GB false we are running pgsync process as --daemon ( as --daemon was crashing with nohup ). Is there a way to solve this issue?

From AWS documentation, I read the following: Replication slots can be created as part of logical decoding feature of AWS Database Migration Service (AWS DMS). For logical replication, the slot parameter rds.logical_replication is set to 1. Replication slots retain the WAL files until the files are externally consumed by a consumer.

I tried to set below env variables with the following values but diskspace is keep on getting full overnight. It looks like cleanup interval setting is not working.

REPLICATION_SLOT_CLEANUP_INTERVAL: 30 POLL_TIMEOUT: 30

Could you please help me with the above issue as I dont know whatelse can I do?

toluaina commented 1 year ago

Can you please provide more deatils. e.g postgres version, pgsync version,

Also, can you run in an interactive mode and set the REPLICATION_SLOT_CLEANUP_INTERVAL to a low value like 5 sec. Do you see the logs for the replication slots getting cleaned up.

Most important: are you able to upgrade to the latest version of PGSync? i.e 2.5.0