search

dask / distributed

A distributed task scheduler for Dask
https://distributed.dask.org
BSD 3-Clause "New" or "Revised" License
1.58k stars 720 forks source link

Specify different resources per worker in the hostfile #870

Open jmontoyam opened 7 years ago

jmontoyam commented 7 years ago

Hi,

I'm a new dask-distributed user. I would like to setup the network using the dask-ssh command and the corresponding hostfile:

dask-ssh --hostfile hostfile.txt

Let's say the workers have different resources constraints, therefore, I would like to specify different resources per worker in the hostfile, for instance:

10.121.0.1 --nprocs 1 --nthreads 4 10.121.0.2 --nprocs 1 --nthreads 2 10.121.0.3 --nprocs 1 --nthreads 1 10.121.0.4 --nprocs 1 --nthreads 8 10.121.0.5 --nprocs 1 --nthreads 2 10.121.0.6 --nprocs 1 --nthreads 1 10.121.0.7 --nprocs 1 --nthreads 1 10.121.0.8 --nprocs 1 --nthreads 16 10.121.0.9 --nprocs 1 --nthreads 4 10.121.0.10 --nprocs 1 --nthreads 1

Can I specify different resources per worker in the hostfile and then call dask-ssh --hostfile hostfile.txt?

Thank you very much for your help.

mrocklin commented 7 years ago

Seems like a reasonable thing to add. You would need to parse keywords a bit and then change around some code in distributed/deploy/ssh.py. Is this something you're interested in contributing?

On Sun, Feb 12, 2017 at 4:00 PM, jmontoyam notifications@github.com wrote:

Hi,

I'm a new dask-distributed user. I would like to setup the network using the dask-ssh command and the corresponding hostfile:

dask-ssh --hostfile hostfile.txt

Let's say the workers have different resources constraints, therefore, I would like to specify different resources per worker in the hostfile, for instance:

10.121.0.1 --nprocs 1 --nthreads 4 10.121.0.2 --nprocs 1 --nthreads 2 10.121.0.3 --nprocs 1 --nthreads 1 10.121.0.4 --nprocs 1 --nthreads 8 10.121.0.5 --nprocs 1 --nthreads 2 10.121.0.6 --nprocs 1 --nthreads 1 10.121.0.7 --nprocs 1 --nthreads 1 10.121.0.8 --nprocs 1 --nthreads 16 10.121.0.9 --nprocs 1 --nthreads 4 10.121.0.10 --nprocs 1 --nthreads 1

Can I specify different resources per worker in the hostfile and then call dask-ssh --hostfile hostfile.txt?

Thank you very much for your help.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dask/distributed/issues/870, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszNSvwQeKaSkufaq3tUUHrXJllwAJks5rb3LvgaJpZM4L-nUi .

jmontoyam commented 7 years ago

Yes!, sure!, I would like to make my first contribution to this amazing project!, but please be patient with me if I ask you too many questions, I'm a beginner dask-distributed user :-).