search

toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.1k stars 172 forks source link

DELETE not working #513

Closed carlos-pereira-costoya closed 5 months ago

carlos-pereira-costoya commented 6 months ago

PGSync version: 3.0.0

Postgres version: 16

Elasticsearch/OpenSearch version: 1.2.0 (OpenSearch

Redis version: 7.2

Python version: 3.9

Problem Description:

Hi, we are running local tests with the example docker-compose from toluana GitHub repository, using both non-partitioned and partitioned tables in PostgreSQL. When performing INSERTs and UPDATEs, the PGSync service detects the changes perfectly. However, when executing DELETEs, it is not reflecting the changes in OpenSearch. The DETACH command is also not working on partitioned tables.

Interestingly, deleting the index in OpenSearch and rerunning PgSync successfully syncs the deleted records to OpenSearch. Is this a known bug, or is there a missing parameter in the configuration?

Error Message (if any):

toluaina commented 6 months ago

Are you able to test this with OpenSearch 2.11.1? Can you also confirm that you ran bootstrap successfully? are there any errors in the terminal? finally, can you run off the latest master branch and confirm is this happens?

carlos-pereira-costoya commented 5 months ago

Hi @toluaina ,

I retested in OpenSearch 2.11, performed a complete load, and cleaned the replication slots in PostgreSQL.

The issue was that I had replication slots with old tests in the Docker environment, and in the cloud environment, I didn't have any volume available to create the checkpoint files. In these cases, the pgsync synchronization wasn't working properly and the delete was nott working.

Thanks for your help, it's much appreciated.

carlos-pereira-costoya commented 5 months ago

Solved, I close the issue