search

OptimalBits / bull

Premium Queue package for handling distributed jobs and messages in NodeJS.
Other
15.48k stars 1.43k forks source link

Advanced rate limiter #1975

Closed gilshallem closed 3 years ago

gilshallem commented 3 years ago

Hi, I'm building a website scraper. I use different proxies to scrape the website so I need to define a rate of x requests per minute per proxy. The only way I can think of is implementing a queue per proxy. is there a better way? in addition, some of the requests need to happen from a specific proxy, yet some can be called from any proxy so this solution will not work.

Thank you for the awesome library!

gilshallem commented 3 years ago

maybe if there is a way for the processor to bounce back the job manually?

manast commented 3 years ago

You can try the groupKey option in the rate limiter: https://github.com/OptimalBits/bull/blob/develop/REFERENCE.md#queue

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

refo commented 3 years ago

Hi, I'm building a website scraper. I use different proxies to scrape the website so I need to define a rate of x requests per minute per proxy. The only way I can think of is implementing a queue per proxy. is there a better way? in addition, some of the requests need to happen from a specific proxy, yet some can be called from any proxy so this solution will not work.

Thank you for the awesome library!

Did you manage to group jobs per proxy without adding a queue per proxy?