Dashboard 404 With Latest Dask/Distributed

[lucasdurand]

I'm having issues accessing the Bokeh Dashboard with the current version of dask-yarn/dask/distributed. I'm expecting to find the dashboard (according to cluster.dashboard_link at /status) but only receive a 404. On the other hand I can navigate to /info/main/workers.html and get referred to the individual dashboard for each worker. In this case I am slammed with a 403.

Reverting to dask-yarn==0.7.0, dask==2.3.0 fixes this.

Reproducing

I'm trying to use a minimal install (although it got a little away from that) with the following Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
venv-pack = "*"

[packages]
dask-yarn = "==0.8.1"
numpy = "*"
dask = {extras = ["dataframe"],version = "==2.15.0"}
pandas = "*"
bokeh = "*"
msgpack = "<1"
jupyter-server-proxy = "*"
pyarrow = "*"
ipython = "*"
distributed = "==2.15.0"
tornado = "<6.0"

[requires]
python_version = "3.7"

I'm packing this up with venv-pack -o dask_env.tar.gz and launching a cluster with

from dask.yarn import YarnCluster
cluster = YarnCluster(
    environment = "dask_env.tar.gz",
    deploy_mode="local",
    worker_memory=...,
    worker_env={...},
    worker_vcores=...
)

Tracebacks

Nothing interesting coming through on an application level, and silence with the 404 for the main dashboard. With the worker dashboards (/proxy/.../............/status) I'm getting:

Traceback (most recent call last):
  File ".../.local/share/virtualenvs/dask_env_current-Hn1YkE44/lib/python3.7/site-packages/tornado/web.py", line 1569, in _execute
    result = self.prepare()
  File ".../.local/share/virtualenvs/dask_env_current-Hn1YkE44/lib/python3.7/site-packages/notebook/base/handlers.py", line 493, in prepare
    raise web.HTTPError(403)
tornado.web.HTTPError: HTTP 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File ".../.local/share/virtualenvs/dask_env_current-Hn1YkE44/lib/python3.7/site-packages/tornado/web.py", line 1112, in send_error
    self.write_error(status_code, **kwargs)
  File ".../.local/share/virtualenvs/dask_env_current-Hn1YkE44/lib/python3.7/site-packages/notebook/base/handlers.py", line 576, in write_error
    html = self.render_template('%s.html' % status_code, **ns)
  File ".../.local/share/virtualenvs/dask_env_current-Hn1YkE44/lib/python3.7/site-packages/notebook/base/handlers.py", line 506, in render_template
    template = self.get_template(name)
  File ".../.local/share/virtualenvs/dask_env_current-Hn1YkE44/lib/python3.7/site-packages/notebook/base/handlers.py", line 502, in get_template
    return self.settings['jinja2_env'].get_template(name)
KeyError: 'jinja2_env'

Version information

- Python version 3.7
- Dask-Yarn version 0.8.1
- Hadoop version, and distribution (guessing this isn't important here) 2.6? I can check again.

[lucasdurand]

I'm back. Looks like everything actually works for everything up to and including:

• dask-yarn==0.8.1 (current)
• dask==2.14.0 (one back)
• distributed==2.14.0 (;))

So much less of an issue than I may have implied.

[TomAugspurger]

@lucasdurand can you try with distributed>=2.15.2 (released earlier today). 2.15.1 and 2.15.2 have a couple changes that should fix this issue.

[lucasdurand]

Works great! Thanks!

[AndreaGiardini]

Hi everyone

I find myself with the same error, but a different stack. I use dask-kubernetes and I have the latest dask and distributed installed. Dask dashboard works fine, but when I try to open the dashboard of one of the workers, I get a 403 from tornado on dask-scheduler

tornado.application - ERROR - Uncaught exception in write_error
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1681, in _execute
    result = self.prepare()
  File "/opt/conda/lib/python3.8/site-packages/notebook/base/handlers.py", line 498, in prepare
    raise web.HTTPError(403)
tornado.web.HTTPError: HTTP 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/tornado/web.py", line 1217, in send_error
    self.write_error(status_code, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/notebook/base/handlers.py", line 581, in write_error
    html = self.render_template('%s.html' % status_code, **ns)
  File "/opt/conda/lib/python3.8/site-packages/notebook/base/handlers.py", line 511, in render_template
    template = self.get_template(name)
  File "/opt/conda/lib/python3.8/site-packages/notebook/base/handlers.py", line 507, in get_template
    return self.settings['jinja2_env'].get_template(name)
KeyError: 'jinja2_env'

I use dask-gateway to create the cluster. Do you have any idea?