problem with using multiple queues

[gregorleban]

What version of async are you using?

v2.0.1

Which environment did the issue occur in (Node version/browser version)

Node v4.2.4

What did you do? Please include a minimal reproducable case illustrating issue.

I have multiple web services (S1, S2, ...) that I need to call with the same http.requests R1, R2, R3, .... Requests have to be processed in the same order as created (first R1, then R2, ...). If one of the web services Sx goes offline, the requests Rx have to wait and then be executed when the web service gets back online.

To achieve this functionality I use the async.queue and use concurency = 1. I create a separate queue for each web service. For each web request that I get I create a task for each web service and enqueue it in the appropriate queue. The callback for the queue is called, which calls the web service. If the request is processed ok, I simply call the callback that I guess removes the task from the queue. In case the request timeouts I call queue.unshift(task) which puts the task back to the beginning of the queue so that it should be called again, and then again call the callback.

The problem I see is that if one web service goes offline, the task in that queue is tried repeatedly, but unfortunately no tasks from the other queues (for web services that are online) are being processed. Is this expected? I would expect that tasks from each queue would be processed independently.

Here is some (pseudo) code that I have:

        // this is how i create a queue for each web service I have
    service.queue = async.queue(function (task, callback) {
        processTaskRequest(task, service, type, callback);
    }, 1);

.....

function processTaskRequest(task, service, type, callback) {
    makeRequest(options, task.req, task.res,
        function (data) {
            if (callback) { callback(); }
        },
        function (e, e2) {
            console.log("Failed task for service ");
            // re-enqueue the task at the beginning of the queue - the same task should be executed next
            // this will make sure all tasks will execute and they will execute in the same order as came in
            service.queue.unshift(task);
            // only after X seconds try to repeat making the query - if service is offline, it doesn't make sense to do it more often
            setTimeout(function () {
                if (callback) { callback(); }
            }, 10000);
        }
    );
} 

Thank you in advance for any help!

[megawac]

Though unshifting back to the top should work it seems a bit sketchy to me. I would probably use this workflow with a mix of async.queue and async.retry

function handler(task, callback) {
   retry({retry-options}, callApiWithTask, function(err, data) {
      callback(data);
   } 
}

let q = queue(handler, 1);

// add task to call first api end point
// add second,
// etc etc.

Not sure exactly what is going wrong your code, it's hard to debug with that pseudo code

[gregorleban]

I didn't know about the retry. I've implemented it and it works perfectly fine. Thank you for the suggestion!

[megawac]

:tada: