verida / data-connector-server

1 stars 1 forks source link

Update Youtube to use `ItemsRangeTracker` #92

Closed tahpot closed 3 weeks ago

tahpot commented 4 weeks ago

I have created a new method of syncing data that is more flexible and ensures that new items are fetched before older items are backfilled.

It users a helper library called ItemsRangeTracker. It's designed to load and save the current position information in syncPosition.thisRef.

I have updated the gmail handler to use this impementation, you can see the diff here.

The flow is like this:

  1. Load the range tracker using syncPosition.thisRef
  2. Get the next range information which will always be configured to fetch any new items, up to the previous item that was fetched.
  3. Use the range information to populate the page token
  4. Update the range tracker with the result of the processing
  5. If not a full batch of results returned, backfill to get more
  6. Again, update the range tracker
  7. Save the range tracker information to syncPosition.thisRef

You'll notice this has simplified the code and there is no longer setNextPosition() or stopSync() methods.

Also see how the unit tests have been updated: https://github.com/verida/data-connector-server/pull/91/files#diff-3878c5244bf405ac39a2ccd4cc0c0df958f2fc5481a71c76a40d9042a658e7da