Randomize queue?

[catdevnull]

I have a program that first gets a bunch of links to download from different hosts, and then adds them to the queue to download. Something like

for (const listLink of lists) {
  fetch(listLink).then(links => links.forEach(link => queue.push(link)))
}

I would like to randomize the queue to distribute the load amongst hosts to maximize the download speeds. One way of doing this is having many queues, one for each host, but the problem is that depending on the N of hosts and M of concurrent tasks per queue, N*M might be too high and drop a bunch of connections and stuff.

Is there a simple way of doing this with fastq? Or is there another package that allows something like this?

[mcollina]

That problem is beyond what fastq is meant to solve.

I'd recommend you to take a look at undici.request and BalancedPool dispatcher.

[catdevnull]

Thank you! That seems very useful. However, unless I'm getting it wrong, it seems that BalancedPool is for N upstreams that all answer the same "kinds" of request. I'm downloading files from a bunch of different servers and services. Am I supposed to have a BalancedPool for each? That would put me back on the same problem, no?

[mcollina]

Unfortunately yes, I totally misunderstood your problem.