singer-io / tap-shopify

Singer.io tap for extracting Shopify data

GNU Affero General Public License v3.0

57 stars 89 forks source link

update bookmarking for interrupted sync #166

Closed RushiT0122 closed 1 year ago

RushiT0122 commented 1 year ago

Description of change

Consider we have following records to sync from the source,

(order_id | replication_key):

(1, 101),
(2, 102),
(3, 103),
(4, 104),
(5, 105),
(6, 106),
(7, 107),
(8, 108),
(7, 109), (9, 109), --> order_id#7 gets updated here
(10, 110)

Before sync starts,

Sync starts at: 110
Window size: 5
Bookmark=100
First window: 100-105

Scenario

Sync interrupted at: since_id=8 (replication_key=108)
Next sync starts at: 115
Record updated before next sync: order_id=7 (replication_key=109)

On completion of next sync,

since_id = 7
Windows: (100-105), (105-110), (110-115)
Bookmark = min(max_bookmark_value, stop_time) = 110

In this scenario, on resuming sync updated record id#7 with (replication_key=109) will not sync with current implementation.

Solution:

Add updated_at_max value along with since_id and on resuming interrupted sync set window between (bookmark, updated_at_max).

Sync interrupted at: since_id=8 (replication_key=108)
Next sync starts at: 115
Record updated before next sync: order_id=7 (replication_key=109)

On completion of next sync,

since_id = 1 --> pop() the since_id=8 from the bookmark
Starting window: (105-108), (108-113), (113-115)
Bookmark = min(max_bookmark_value, stop_time) = 110

This solution will make sure on resuming the interrupted sync, record id#7 (replication_key=109) will be synced.

QA steps

[x] automated tests passing
[ ] manual qa steps passing (list below)

Risks

Rollback steps

revert this branch