sindresorhus / p-queue

Promise queue with concurrency control
MIT License
3.39k stars 182 forks source link

Feature Request - awaitNextInterval #205

Open jaysteiner-dev opened 4 months ago

jaysteiner-dev commented 4 months ago

I'm using the p-queue for throttling, where potentially the API I am trying to hit could still rate limit me. It would be great if there was a function I could use to handle those timeout after I catch the errors.

const queue = new PQueue({
    concurrency: 50,
    interval: 10000,
    intervalCap: 1000
    carryoverConcurrencyCount: true
});

const promises = myObjects.map(
  (obj) => async () =>
    myApiCall(obj.id).catch((err) => {
      if (err.code === HttpStatus.TOO_MANY_REQUESTS) {
        queue.awaitNextInterval()
      }
      this.logger.error(`failed with:`, err);
    })
);

await queue.addAll(promises);
sindresorhus commented 4 months ago

What exactly would queue.awaitNextInterval() do?

gridagribnoy commented 1 month ago
const promises = myObjects.map(obj => {
  const cb = async () => {
    return myApiCall(obj.id).catch(err => {
      if (err.code === HttpStatus.TOO_MANY_REQUESTS) {
        //queue.awaitNextInterval()
        return queue.add(cb, { priority: 999999 });
      }
      this.logger.error(`failed with:`, err);
    });
  };
  return cb;
});

await queue.addAll(promises);