search

t3chnoboy / amazon-product-api

:credit_card: Amazon Product Advertising API client
365 stars 104 forks source link

Request Throttling #37

Closed FdezRomero closed 7 years ago

FdezRomero commented 8 years ago

Hi all!

This is a question rather than an issue: What is in your opinion the best way to throttle requests to the Amazon Product API? The initial limit is 1 req/s, increasing by 1 req/s for every $4,700 in goods sold by your awsTag.

I'm currently using Bottleneck with a limit of 1 request per 2000ms and I'm still getting many RequestThrottled errors. So it doesn't seem to be playing well with this module or the API.

Are any of you throttling your requests? How? It would be great if this feature was included, as similar projects in other languages have a MaxQPS option.

Thanks!

t3chnoboy commented 8 years ago

Hey @FdezRomero! It think it is a good idea to add an option to limit requests. https://github.com/jhurliman/node-rate-limiter comes to mind. Any advantages of using bottleneck over it? Can you show an example of that option in other languages?

FdezRomero commented 8 years ago

Hi @t3chnoboy,

I chose Bottleneck over node-rate-limiter because it allows to create a different keys/queues for each API endpoint and seemed very lightweight and performant (the creators use it for production DNS). However, I'm not sure if some endpoints are counted together for the rate limiting or not, I found contradictory information about this. Anyway, I have no preference in using these libraries or a custom implementation as long as it supports concurrency.

You can find an example of the MaxQPS I mentioned before in bottlenose, a client written in Python (I haven't used it myself).

Thanks!

kaijuliu commented 8 years ago

Hi @FdezRomero. I recently ran into the same throttling issue and found that global throttling, i.e. not per-endpoint throttling, was the solution. Putting https://github.com/jhurliman/node-rate-limiter, as @t3chnoboy suggested, in front of all requests worked well.

FdezRomero commented 8 years ago

Thanks @kaijuliu, knowing that the limit is actually global and not per endpoint really helps :+1: I read some time ago a comment saying that the European stores acted like a cluster and the rest were independent, but I never found official information from Amazon...

masterT commented 7 years ago

@t3chnoboy can we close this issue?