Dashboard not found with dask 2.15 & dask-jobqueue 0.71

dask / distributed

A distributed task scheduler for Dask

https://distributed.dask.org

BSD 3-Clause "New" or "Revised" License

1.58k stars 718 forks source link

Dashboard not found with dask 2.15 & dask-jobqueue 0.71 #3748

Closed fbriol closed 4 years ago

fbriol commented 4 years ago

Hello,

The dashboard is no longer accessible when using dask v2.15 with dask-jobqueue. This works with the local cluster, but does not work (error code 404) on the cluster. I reinstalled version 2.14 and the dashboard is functional again. I am available to give you more information. Do not hesitate to contact me.

TomAugspurger commented 4 years ago

This may be fixed by https://github.com/dask/distributed/pull/3746. If you're able to test out distributed master that be great.

jrbourbeau commented 4 years ago

Thanks for raising an issue @fbriol! There was a recent dashboard-related fix that's been merged here, but has not yet been released. Would you be willing to try the latest development version of distributed to see if the issue you're seeing has been resolved?

fbriol commented 4 years ago

This works properly with the master branch.

jrbourbeau commented 4 years ago

Awesome, thanks for trying it out! There will be a new release of distributed today which will include the dashboard fix

jrbourbeau commented 4 years ago

Closing this issue as distributed==2.15.1 is out on PyPI and conda-forge. Thanks again @fbriol

fbriol commented 4 years ago

It now works with jupyterlab and jupyter-server-proxy (the dashboard is not directly accessible on the cluster), however, indirect access (http://hostname:8787) I still have error 404.

rodgerduffett commented 4 years ago

I am also still seeing this. A fresh installation of miniconda with dask 2.15.0 dask-jobqueue 0.7.1 dask-labextension 2.0.2 distributed 2.15.1 shows error 404 when accessing any dashboard pages on port 8787. Reverting distributed to 2.14.0 fixes the problem.

TomAugspurger commented 4 years ago

@rodgerduffett are you able to try out https://github.com/dask/distributed/pull/3758 by chance?

rodgerduffett commented 4 years ago

Hi Tom, Going back to distributed 2.15.1 and applying the patch to distributed/scheduler.py from #3758 fixes this problem. I am able to access all dashboard pages on port 8787. Thank-you for your help!

TomAugspurger commented 4 years ago

Thanks for verifying. I'm hoping to have that in today, and will probably do a 2.15.2 release with it.

fbriol commented 4 years ago

It works with the patch, however, I have this error now. Any ideas?

HTTPServerRequest(protocol='http', host='xxx.fr', method='GET', uri='/status/ws', version='HTTP/1.1', remote_ip='127.0.0.1')
Traceback (most recent call last):
  File "/home/ad/briolf/odatis/briolf/anaconda3/lib/python3.7/site-packages/tornado/websocket.py", line 956, in _accept_connection
    open_result = handler.open(*handler.open_args, **handler.open_kwargs)
  File "/home/ad/briolf/odatis/briolf/anaconda3/lib/python3.7/site-packages/bokeh/server/views/ws.py", line 123, in open
    raise ProtocolError("Subprotocol header is not 'bokeh'")
bokeh.protocol.exceptions.ProtocolError: Subprotocol header is not 'bokeh'

fbriol commented 4 years ago

This is a jupyter-server-proxy error, corrected in this issue: https://github.com/jupyterhub/jupyter-server-proxy/pull/180

jrbourbeau commented 4 years ago

Thanks @fbriol @rodgerduffett for helping out here. distributed==2.15.2 is out which includes the fixes in https://github.com/dask/distributed/pull/3758

andersy005 commented 4 years ago

I am getting a slight different error:

Has anyone seen this? I am running the following versions:

In [1]: import dask, distributed, bokeh

In [2]: dask.__version__, distributed.__version__, bokeh.__version__
Out[2]: ('2.16.0', '2.16.0', '2.0.1')