long2ice / meilisync

Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch
https://github.com/long2ice/meilisync
Apache License 2.0
260 stars 40 forks source link

Meilisync loses event when both create and update occur for the same primary key in an interval #92

Closed rishabhc32 closed 1 month ago

rishabhc32 commented 6 months ago

If I have an interval of, let's suppose, 10 seconds and insert size of 1000 documents, and within that interval, a create event is first received followed by an update event for the same primary key, the current implementation of Meilisync's event buffer leads to the create event being overwritten by the update event.

https://github.com/long2ice/meilisync/blob/052c4de9ab4e7ed91285b981c7ab5beeb03e90ec/meilisync/event.py#L10-L13

self._events[sync][pk] = event here the event will get overwritten.

A potential solution could involve modifying the event handling logic to append events rather than overwrite them, for instance:

self._events[sync][pk].append(event)

Or we could just do:

self._events[sync].append(event)
rishabhc32 commented 4 months ago

@long2ice Would you be interested in a PR where a solution like self._events[sync][pk].append(event) is implemented? This will preserve the order of the events on a primary key.

KDMW-IO commented 1 month ago

I had included my fix for this on https://github.com/long2ice/meilisync/pull/103