singer-io / tap-shopify

Singer.io tap for extracting Shopify data

GNU Affero General Public License v3.0

56 stars 88 forks source link

TDL-17096 updated the write_bookmark() logic #143

Closed namrata270998 closed 1 year ago

namrata270998 commented 2 years ago

Description of change

Updated the bookmark written for base streams.
- bookmark on the lesser of the maximum replication key value and the start time of sync execution.
Added unittests for the same

QA steps

Check that the correct bookmark is written based on min-max evaluation
was pr-alpha'd to two separate clients, but without feedback if it fixed their data discrepancies.

Risks

the tap will likely take longer with extra api calls because of the new bookmarking. performance testing was done in this card: https://jira.talendforge.org/browse/TDL-17096
Rollback steps
- revert this branch