search

toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.16k stars 180 forks source link

Fix missing delta inserts of first-level children #493

Open mstdokumaci opened 1 year ago

mstdokumaci commented 1 year ago

When pgsync is catching up to changes via PG_NOTIFY or replication slots, it is missing inserts of first-level children. This commit is fixing the issue.

dorin-musteata commented 1 year ago

Hi, what is meant by first level children ?

Suppose I have a root table called product and categories which got a product_id reference. By first level children do we mean inserts on categories table ?

Asking because I have this model and indeed if insert is happening on categories, product document is not updated with new category.

mstdokumaci commented 1 year ago

Hi, what is meant by first level children ?

Suppose I have a root table called product and categories which got a product_id reference. By first level children do we mean inserts on categories table ?

Asking because I have this model and indeed if insert is happening on categories, product document is not updated with new category.

yes, it only happens at first level children as you described and only on new inserts after pgsync has started. if you delete the checkpoint file and pgsync starts with a full sync, it will update all documents correctly.

dorin-musteata commented 1 year ago

Hi, what is meant by first level children ? Suppose I have a root table called product and categories which got a product_id reference. By first level children do we mean inserts on categories table ? Asking because I have this model and indeed if insert is happening on categories, product document is not updated with new category.

yes, it only happens at first level children as you described and only on new inserts after pgsync has started. if you delete the checkpoint file and pgsync starts with a full sync, it will update all documents correctly.

Doesn't seem to make a difference, if child record is inserted, parent document is not updated.

I tested it with one (product) to many (categories) relationship.

Maybe we are having two different issues, for which I have to open a bug ticket.

Thanks anyway 👍

Ziming-W commented 9 months ago

I encountered the same issue. But your commit doesn't seem to solve the problem for me. In my local pgsync library, I manually replaced the file you changed, the issue persists. Maybe I did it the wrong way?