Deploying dask using Docker on different machines

simplew2011 commented 4 months ago

I have three machines, one scheduler machines and two worker machines

scheduler machines (ip: 192.168.30.161)，start scheduler

sudo docker run -it -p 8786:8786 -p 8787:8787 --name=dask_scheduler dataprocess:v0.1.0 dask scheduler

worker1 machines (ip: 192.168.20.139)，start worker

sudo docker run -it --name=dask_worker dataprocess:v0.1.0 dask worker 192.168.30.161:8786 --nworkers 64 --nthreads 1 --memory-limit 512GB

worker2 machines (ip: 192.168.25.141)，start worker

sudo docker run -it --name=dask_worker dataprocess:v0.1.0 dask worker 192.168.30.161:8786 --nworkers 64 --nthreads 1 --memory-limit 512GB

scheduler machines，start dask to load data and process

sudo docker run -it  --name=do_task  dataprocess:v0.1.0 python process_data.py

process_data.py

import dask.dataframe as dd
from dask.distributed import Client

if __name__ == '__main__':
    client = Client('192.168.30.161:8786') # error
    # client = Client(n_workers=16, threads_per_worker=1) # ok
    dataset_dask = dd.read_json('/mnt/temp/demos/datasets.jsonl', lines=True, blocksize='10MB')
    print(len(dataset_dask.persist()))

error in scheduler

2024-05-30 11:21:15,555 - distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {"('len-agg-54c9156b5d83b7d7417c7c710d5109d4', 0)": ('tcp://127.0.0.1:32839',)}
2024-05-30 11:21:45,941 - distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {"('len-agg-54c9156b5d83b7d7417c7c710d5109d4', 0)": ('tcp://127.0.0.1:33649',)}

and some error occur in worker:

2024-05-30 11:25:15,461 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:34869'. Reason: scheduler-close
2024-05-30 11:25:15,462 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:42627'. Reason: scheduler-close
2024-05-30 11:25:15,462 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,462 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,462 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,463 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,463 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,463 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,464 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,464 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,464 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,464 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,465 - distributed.nanny - INFO - Worker closed
2024-05-30 11:25:15,468 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:37473'. Reason: scheduler-close
2024-05-30 11:25:15,468 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:44521'. Reason: scheduler-close
2024-05-30 11:25:15,468 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:42897'. Reason: scheduler-close
2024-05-30 11:25:15,469 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:40241'. Reason: scheduler-close
2024-05-30 11:25:15,469 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:36871'. Reason: scheduler-close
2024-05-30 11:25:15,469 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:42959'. Reason: scheduler-close
2024-05-30 11:25:15,470 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:39851'. Reason: scheduler-close
2024-05-30 11:25:15,470 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:46271'. Reason: scheduler-close
2024-05-30 11:25:15,470 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:41039'. Reason: scheduler-close
2024-05-30 11:25:15,471 - distributed.nanny - INFO - Closing Nanny gracefully at 'tcp://127.0.0.1:45055'. Reason: scheduler-close

and dashboard is empty, can not found status and workers info
if using a LocalCluster, process_data.py is run ok in docker
```
client = Client(n_workers=16, threads_per_worker=1)
```
If I don't use Docker, starting the dask scheduler and two workers using console commands on different machines, and then running the business code is also okay
Is there any problem with my dockers startup command?
- reference #1281 ，set --interface lo in docker run, some error; I'm not sure what parameters the --host needs to set
- run ifconfig in three machines, find the same network interface lo

mrocklin commented 4 months ago

cc @jacobtomlinson maybe you have time to help here

simplew2011 commented 4 months ago

simplew2011 commented 3 months ago

if i add --ipc=host --network host in every docker run, process_data.py is run ok
but the dashboard page is still blank

2024-05-31 08:08:34,530 - distributed.scheduler - INFO -   Scheduler at: tcp://192.168.30.161:8786
2024-05-31 08:08:34,530 - distributed.scheduler - INFO -   dashboard at:  http://192.168.30.161:8787/status
2024-05-31 08:10:16,884 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://192.168.20.139:43259', status: init, memory: 0, processing: 0>
2024-05-31 08:10:17,174 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.20.139:43259
2024-05-31 08:10:17,175 - distributed.core - INFO - Starting established connection to tcp://192.168.20.139:43478
2024-05-31 08:10:17,178 - distributed.scheduler - INFO - Register worker <WorkerState 'tcp://192.168.20.139:37555', status: init, memory: 0, processing: 0>

jacobtomlinson commented 3 months ago

Dask workers connect to each other on random high ports, so when you run the workers without any network or port settings they cannot talk to each other.

Setting --network host will work because it removes network isolation. Alternatively you can create a dedicated docker network which will allow all the containers to talk to each other, this is what we do in our documentation examples.

docker network create dask

docker run --network dask -p 8787:8787 --name scheduler ghcr.io/dask/dask dask scheduler  # start scheduler

docker run --network dask ghcr.io/dask/dask dask worker scheduler:8786 # start worker
docker run --network dask ghcr.io/dask/dask dask worker scheduler:8786 # start worker
docker run --network dask ghcr.io/dask/dask dask worker scheduler:8786 # start worker

Following these steps everything works and I am unable to reproduce the dashboard issue with the latest version of the ghcr.io/dask/dask image.

dask / distributed

Deploying dask using Docker on different machines #8658