search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.76k stars 4.04k forks source link

🐛 Source Shopify: not acknowledging higher Shopify plus rate limits #32818

Open tejastrzinar opened 10 months ago

tejastrzinar commented 10 months ago

Connector Name

source-shopify

Connector Version

1.1.4

What step the error happened?

Other

Relevant information

Extracting time from Shopify is slower than needed on the Shopify Plus plan.

Because of Shopify's rate limits, there is a rate limiter code in the source that introduces a wait time between API calls.

Currently, the lowest wait time (on_low_load) is 0.2 seconds (200 milliseconds). In reality, this produces about 2.5 requests/second, which is 8x lower than what Shopify Plus offers (20 requests/second).

Would it be possible to introduce an even lower wait time to make the extraction of data from Shopify shops with lots of traffic more manageable?

Relevant log output

No response

Contribute

tejastrzinar commented 9 months ago

I opened a pull request for this open issue here: https://github.com/airbytehq/airbyte/pull/33575