taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
6.18k stars 406 forks source link

Rate limiting doesn't work as expected #1649

Closed flaviolivolsi closed 1 year ago

flaviolivolsi commented 1 year ago

I created a rate limited worker setting a max concurrency of 70 workers every 60 seconds, but the worker will process just one job at a time and put the rest in waiting list. Am I missing something? I'm using bullmq 3.5.5.

This is the worker code:

const worker = new Worker(
  'Songs',
  async (job) => generate(job),
  {
    limiter: {
      max: 70,
      duration: 60000,
    },
    connection,
  }
)
manast commented 1 year ago

You are not configuring the concurrency with the limiter, only the max allowed number of jobs per unit of time. In your case, if your job is slower than 60000/70 the worker will process less than what you have specified in the limiter. Try with a larger concurrency factor (will only help if your jobs are IO bound, if not, you will need more processes with workers):

const worker = new Worker(
  'Songs',
  async (job) => generate(job),
  {
    limiter: {
      max: 70,
      duration: 60000,
    },
    concurrency: 100,
    connection,
  }
)
flaviolivolsi commented 1 year ago

Thank you @manast and sorry for the oversight - it works perfectly now!