Currently we Promise.all() batches of concurrent jobs (default is 20 at once).
Say we have two slow jobs - A and B in a set of 30 jobs that each take 10 seconds to run, whereas most jobs execute in under a second. If A ends up in the first batch of 20 whilst B ends up in the second batch of twenty, then we will end up waiting for 20+ seconds in total. This is because we are always limited by the slowest job in a batch.
If instead, we don't batch but just limit the number of concurrent jobs, then we can speed up await driver.jobsComplete() calls in cases where a few bad actors slow down the whole run.
Currently we
Promise.all()batches of concurrent jobs (default is 20 at once).Say we have two slow jobs -
AandBin a set of 30 jobs that each take 10 seconds to run, whereas most jobs execute in under a second. IfAends up in the first batch of 20 whilstBends up in the second batch of twenty, then we will end up waiting for 20+ seconds in total. This is because we are always limited by the slowest job in a batch.If instead, we don't batch but just limit the number of concurrent jobs, then we can speed up
await driver.jobsComplete()calls in cases where a few bad actors slow down the whole run.