Troy has been running into issues with slow worker initialization causing problems with the client.scatter call. Because the workers are not yet visible to the main runner, the client.scatter fails. This is a known issue with dask distributed (https://github.com/dask/distributed/issues/2941). We can explicitly write out the large side inputs to disk (or pickle them), and send along file references that can be decoded on the workers.
This is more of a problem with SLURM distributed clusters than typical local cluster configurations.
Before submitting
Please check the following:
[x] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
[ ] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
[x] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Bug report
Troy has been running into issues with slow worker initialization causing problems with the
client.scattercall. Because the workers are not yet visible to the main runner, theclient.scatterfails. This is a known issue with dask distributed (https://github.com/dask/distributed/issues/2941). We can explicitly write out the large side inputs to disk (or pickle them), and send along file references that can be decoded on the workers.This is more of a problem with SLURM distributed clusters than typical local cluster configurations.
Before submitting Please check the following: