When starting up, Joblib polls the number of workers in a cluster to determine the number of tasks that can run in parallel. If this happens before all the workers in a cluster arrive, then the number of workers Joblib thinks the cluster have can be a large underestimate, resulting in an under-utilization of the cluster. As a workaround, this PR adds an update to wait for all workers to arrive before using Joblib.
When starting up, Joblib polls the number of workers in a cluster to determine the number of tasks that can run in parallel. If this happens before all the workers in a cluster arrive, then the number of workers Joblib thinks the cluster have can be a large underestimate, resulting in an under-utilization of the cluster. As a workaround, this PR adds an update to wait for all workers to arrive before using Joblib.
(thanks to @mrocklin for noticing this)