When using upload_file to upload all the required modules and JSON files to 2 worker nodes, it takes around 80 seconds. This is likely to increase proportional to the number of workers. This data is not dynamic, so the working environment can be pre-existing on each worker node. However, the workers need to know the Python interpreter location and the working directory.
I have tried multiple things to no avail. I have an open issue in Dask Discourse.
When using upload_file to upload all the required modules and JSON files to 2 worker nodes, it takes around 80 seconds. This is likely to increase proportional to the number of workers. This data is not dynamic, so the working environment can be pre-existing on each worker node. However, the workers need to know the Python interpreter location and the working directory.
I have tried multiple things to no avail. I have an open issue in Dask Discourse.