mcollina / fastq

Fast, in memory work queue
ISC License
944 stars 47 forks source link

Waiting for the queue to be fully processed #47

Closed robogeek closed 3 years ago

robogeek commented 3 years ago

Your documentation didn't help me understand how to use the drain function to wait until the queue is fully processed.

What I came up with is:

const queue = fastq.promise(renderDocumentInQueue,
                            config.concurrency);

const waitFor = [];
for (let entry of filez) {
    waitFor.push(queue.push(entry));
}

await Promise.all(waitFor);

This is using the Promise version of fastq. In that version, the push method returns a Promise. Therefore pushing the Promise's into an array lets us use Promise.all to know when all tasks have been completed.

Does this look correct?

An alternative I've coded but haven't tested is:

// await new Promise((resolve, reject) => {
//    queue.drain = function() {
//        resolve();
//    }
// });

It seems from your test scripts that one simply assigns a function to the drain field, and it'll automatically be called. Therefore one way to wait for this function to be called is this Promise here.

Is either preferable over the other?

mcollina commented 3 years ago

This is using the Promise version of fastq. In that version, the push method returns a Promise. Therefore pushing the Promise's into an array lets us use Promise.all to know when all tasks have been completed.

Does this look correct?

Yes