search

dask / dask-yarn

Deploy dask on YARN clusters
http://yarn.dask.org
BSD 3-Clause "New" or "Revised" License
69 stars 41 forks source link

fea/update-worker #140

Closed quasiben closed 3 years ago

quasiben commented 3 years ago

PR replaces usage of Nanny in the worker with dask_spec. Using dask_spec users can can define worker_classes and worker_options in an effort to mimic the style of deployment leveraged in dask-cloudprovider

ayushdg commented 3 years ago

Thanks for this @quasiben. I was able to use this pr to spin up dask-cuda workers using the worker_class argument. Is there any more work needed here before it can be reviewed/merged?

Side question (not directly related to the PR): Is it possible to spin up workers on the main (master) node as well?

quasiben commented 3 years ago

Thanks for the comments @jacobtomlinson !

quasiben commented 3 years ago

Thanks merging in and we'll release shortly