search

dask / dask-yarn

Deploy dask on YARN clusters
http://yarn.dask.org
BSD 3-Clause "New" or "Revised" License
69 stars 41 forks source link

Jupyter notebook widget can't scale cluster #79

Closed dieselcat closed 5 years ago

dieselcat commented 5 years ago

I've got a standard deployment of Dask 2.0 on AWS EMR Cluster with original bootstrap shell script. I can connect to Jupyter Notebook and run the example code:

from dask_yarn import YarnCluster
from dask.distributed import Client

cluster = YarnCluster(environment='environment.tar.gz',
                      worker_vcores=2,
                      worker_memory="3GiB",
                      deploy_mode='local')
client = Client(cluster)
cluster

But when trying to scale up the cluster through widget I get an error:

tornado.application - ERROR - Exception in callback <function YarnCluster._widget..update at 0x7fc78597c378> Traceback (most recent call last): File "/home/hadoop/miniconda/lib/python3.6/site-packages/tornado/ioloop.py", line 907, in _run return self.callback() File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/core.py", line 649, in update status.value = self._widget_status() File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/core.py", line 585, in _widget_status cores = sum(w['ncores'] for w in workers.values()) > File "/home/hadoop/miniconda/lib/python3.6/site-packages/dask_yarn/core.py", line 585, in cores = sum(w['ncores'] for w in workers.values()) KeyError: 'ncores'

App log contains the following warning:

/mnt/yarn/usercache/hadoop/appcache/application_.../container_.../environment/lib/python3.6/site-packages/distributed/nanny.py:102: UserWarning: the ncores= parameter has moved to nthreads=

The problem disappeared after I changed the line 585 to: cores = sum(w['nthreads'] for w in workers.values()) if workers.values() else 0

Environment details:

dask 2.0.0
dask-core 2.0.0
dask-yarn 0.6.1
notebook 5.7.8 tornado 6.0.3 distributed 2.0.1 bokeh 1.2.0 ipykernel 5.1.1
ipython 7.6.0
ipython_genutils 0.2.0
ipywidgets 7.4.2

mrocklin commented 5 years ago

Moving this to dask-yarn. cc'ing @jcrist

quasiben commented 5 years ago

This PR might be relevant: https://github.com/dask/dask-yarn/pull/78

AlJohri commented 5 years ago

Yeah, I'm using my fork until this is merged in (pip install git+https://github.com/AlJohri/dask-yarn.git#egg=dask-yarn)

jcrist commented 5 years ago

Thanks for the issue report, this was fixed in #78. dask-yarn version 0.6.2 has been released with this patch, and is available now on PyPI (and soon on conda-forge). Closing.