singer-io / tap-harvest

A Singer tap for extracting data from the Harvest API
GNU Affero General Public License v3.0
8 stars 13 forks source link

tap sync doesn't include records that match the current bookmark #34

Open HarrisonMarcRose opened 4 years ago

HarrisonMarcRose commented 4 years ago

It is generally expected that the tap will save the latest bookmark and in the next sync include all data with the bookmark value greater than or equal to (>=) that bookmark value.

We created 3 projects and did a sync. all three records were synced If we run a second sync without updating, creating or deleting any projects we would expect that the last project updated (in this case also the last project created) would be sent to the target.

Instead however, 0 records for the projects stream were sent to the target.

Also, it appears that Harvest only stores bookmarks with second precision and not millisecond or greater precision. This means that if someone updates or creates a record within the same second just after a sync has finished there is a decent likelihood that this record would be skipped and there would be a data discrepancy. This so far has not been proven and more testing is required around this to see if this actually happens.