search

python-arq / arq

Fast job queuing and RPC in python with asyncio and redis.
https://arq-docs.helpmanual.io/
MIT License
2.15k stars 174 forks source link

Time based Queue. Handling X jobs per minute. #453

Open waseemhnyc opened 5 months ago

waseemhnyc commented 5 months ago

How could you limit the number of jobs you are processing per a set time frame. For example I'd only like to have 60 jobs run per minute. This is helpful if you'd like to do something where you track the number of API calls you are making to avoid running in to rate limiting issues.

JonasKs commented 5 months ago

There's a few ways to solve this, either by limiting a worker max concurrent jobs, or by using libraries such as redis-rate-limiters.