search

phpclassic / php-shopify

PHP SDK for Shopify API
Apache License 2.0
569 stars 211 forks source link

Automatic throttling of API calls #67

Open warely opened 5 years ago

warely commented 5 years ago

Is it possible to build in an automatic throttling of API calls to avoid the too many requests error? There are many other implementations of the leaky bucket algorithm.

Basically, if the API response is the too many requests error, wait X seconds and retry the request.

tareqtms commented 5 years ago

It's already there.

shawnhind commented 5 years ago

@warely just bare in mind that the library will keep retrying indefinitely in a while loop until it gets a success. This can be a huge problem if you have an application that runs concurrent workers that runs jobs in parallel. It can leave some of your worker processes hanging for indefinite amounts of time waiting for requests to succeed.

I ended up forking this repo to remove the retry logic and implemented it into our queue retry system.

warely commented 5 years ago

@shawnhind Can you expand into what you did to create a queue retry system? I assume the loops can be broken by setting hard call limits for script execution times on the PHP script?

shawnhind commented 5 years ago

@warely well our app is in laravel and we use background jobs and Laravel's queue system to process some shopify data. When an API call fails due to throttling we re-queue the job with a delay. But the queue is set up to be prioritized in such a way that if there are other jobs to process it won't keep attempting ones that are failing. If I had left this libraries throttling code in, all the queue workers who started getting a throttle error would have kept being retried and tying up that worker instead of processing other things.