caolan / async

Async utilities for node and the browser
http://caolan.github.io/async/
MIT License
28.15k stars 2.41k forks source link

Is there any way to limit the size of the async.queue? #1950

Closed iamashay closed 4 months ago

iamashay commented 5 months ago

Is there any way to limit the size of the async.queue such that if items pushed into it exceed the maximum limit, it will throw an error? I have a RabbitMQ queue which has GitHub URLs that would be used to call a function which initializes Docker containers to process (build) them. At max, I want to have 2 containers to be created. That's why I would want the async.queue to throw an error on reaching max limit so that the URL message could go back to the RabbitMQ.

thib3113 commented 4 months ago

@iamashay => you can read .length() on queue ? and so throw if the number returned is too high ?

Also, why not just using rabbitMQ ? tell rabbitMQ you just want two tasks in the same time (prefetch if I remember ?) ? and ack when task finish to get next one ?

iamashay commented 4 months ago

@thib3113 , I read that I could make use of dead letter exchange feature to do this. Since I'm still new to RabbitMQ, I found it difficult to implement. For now, I'm using a semaphore library which takes care of running the function that runs docker instance at most 2 times, according to my requirement.

thib3113 commented 4 months ago

@iamashay you can also use ack / nack ? and so the messages will go back in the current queue ? ( I don't remember setting this specifically ? https://www.rabbitmq.com/docs/nack )