search

Matatika / tap-shopify

This Shopify tap produces JSON-formatted data following the Singer spec.
GNU Affero General Public License v3.0
3 stars 12 forks source link

Fix `RuntimeError: Loop detected in pagination` #5

Closed ReubenFrankel closed 1 year ago

ReubenFrankel commented 1 year ago

Error scenario

An initial request is made to

https:/mock-store.myshopify.com/api/2022-01/checkouts.json

The initial request response contains a next link to

https:/mock-store.myshopify.com/api/2022-01/checkouts.json?limit=50&page_info=2

A second request is made, in theory to

https:/mock-store.myshopify.com/api/2022-01/checkouts.json?limit=50&page_info=2

but actually is

https://mock-store.myshopify.com/api/2022-01/checkouts.json

since the URL parameters were not applied from the previous next link, so the same resource as the initial request ends up being fetched, containing the same next link.

Loop is detected by the SDK and an error is thrown with the message

RuntimeError: Loop detected in pagination. Pagination token mock-store.myshopify.com/api/2022-01/checkouts.json?limit=50&page_info=2 is identical to prior token.

Going forward, we should upgrade this tap to a newer version of the SDK that supports pagination and other similar features to abstract away complexities like this fix addresses.