search

dask / dask-labextension

JupyterLab extension for Dask
BSD 3-Clause "New" or "Revised" License
312 stars 63 forks source link

Failed to connect to Bokeh server Error: Could not open websocket #51

Closed andersy005 closed 5 years ago

andersy005 commented 5 years ago

When I try to use the extension via JupyterHub I get the following websocket error

screen shot 2019-03-07 at 12 30 05 pm

I can confirm that the extension was properly installed:

$ jupyter serverextension list
config dir: /ncar/usr/jupyterhub/20190118/etc/jupyter
    dask_labextension  enabled 
    - Validating...
      dask_labextension 0.3.1 OK
    jupyterlab  enabled 
    - Validating...
      jupyterlab 0.35.4 OK
    nbserverproxy  enabled 
    - Validating...
      nbserverproxy  OK

@jhamman pointed out that he thought the websocket problem was fixed in https://github.com/bokeh/bokeh/issues/6023. I am wondering whether this has to do with dask-labextension or it is something that would need to be addressed upstream in Bokeh.

Ccing @jbaksta as he is the Jhub's admin, and knows more about the Jhub setup than me.

andersy005 commented 5 years ago

Adding this post about similar bokeh error for future reference: https://groups.google.com/a/continuum.io/d/msg/bokeh/3fuiBg0nGPM/m8LrBiVCAQAJ

ian-r-rose commented 5 years ago

@andersy005 Thanks for the report. I'm not sure just now where the problem is. Just to confirm, you find that it works okay on the same cluster without using JupyterHub? Also, cc @yuvipanda, who may have some idea about what might be going wrong in the proxying.

andersy005 commented 5 years ago

Just to confirm, you find that it works okay on the same cluster without using JupyterHub?

This is correct. When using SSH tunneling, everything works okay.

jhamman commented 5 years ago

One diagnostic detail I've found instructive here is that the dask dashboard (when opened via nbserverproxy and not in the lab extension) also suffers from the same problem. My best bet is that we have some firewall setting on the hub ingress controller that is blocking websockets but as I've told @andersy005, this is way outside my area of expertise.

jbaksta commented 5 years ago

@andersy005 asked me to give an update here. When using a reverse proxy (i.e., nginx, apache), the websockets connection upgrade needs to be pretty general and the documented ones ones on JupyterHub for normal functionality are too strict in the sense. I'm guessing this will probably be a thing moving forward with jupyterlab extensions in many cases.

andersy005 commented 5 years ago

Closing this as it's been fixed!

ian-r-rose commented 5 years ago

Thanks for the update @andersy005 and @jbaksta! If either of you have a minute to describe how to fix a problem like this, a PR to the README.md would be much appreciated.

benjaminsingleton commented 5 years ago

I believe I've run into this same issue. While I can start a cluster and generate a valid dashboard URL, the graphs in the panes display nothing. It appears that I cannot connect to the Bokeh server.

My setup may be somewhat unusual / complicated. I'm running JupyterLab in a Docker container that is being presented inside of an iframe (I'm using the Domino Data Lab platform). I only wanted to mention this because there may be configuration settings in Domino that might affect the behavior of this extension. I'm honestly just not sure, so I figured I'd raise the question here in case there is an obvious fix / approach to troubleshooting.

dask-jupyterlab-extension

bokeh-error

$ jupyter serverextension list
config dir: /home/ubuntu/.jupyter
    nbserverproxy  enabled
    - Validating...
      nbserverproxy  OK
config dir: /opt/conda/etc/jupyter
    dask_labextension  enabled
    - Validating...
/opt/conda/lib/python3.6/site-packages/dask_labextension/config.py:13: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for fulldetails.
  defaults = yaml.load(f)
      dask_labextension 0.3.1 OK
    ipyparallel.nbextension  enabled
    - Validating...
      ipyparallel.nbextension  OK
    jupyterlab  enabled
    - Validating...
      jupyterlab 0.35.4 OK
    jupyterlab_latex  enabled
    - Validating...
      jupyterlab_latex 0.4.1 OK
    nbdime  enabled
    - Validating...
      nbdime 1.0.5 OK
ian-r-rose commented 5 years ago

Hi @benjaminsingleton, thanks for the report. This may be the same issue as dask/dask-examples#64. Can you try downgrading notebook to 5.7.4 and tornado to 5.1.1 and see if that helps?

benjaminsingleton commented 5 years ago

Thanks! I will test with the versions you provided and will report back.

benjaminsingleton commented 5 years ago

@ian-r-rose Just wanted to report back that downgrading notebook=5.7.4 and tornado=5.1.1 solved the problem, as you directed! All the graphs are updating as they should.

ian-r-rose commented 5 years ago

Great! We are still dealing with some difficulties with the newest tornado, but once they are nailed down you should be able to update without difficulties.

rabernat commented 5 years ago

I just hit this same issue on nasa pleiades. Has it been resolved, or do I need to downgrade notebook and tornado as recommended?

ian-r-rose commented 5 years ago

What version of dask_labextension are you using? This should be fixed as of dask_labextension 0.3.2, and the jupyterlab compatibility issues should be fixed as of 1.0.0.

ian-r-rose commented 5 years ago

If you need to be on an earlier version than those, I would recommend downgrading tornado to 5.1.1

rabernat commented 5 years ago

I'm on dask-labextension v1.0.0 and jupyterlab 1.0.1.

The problem is identical to the one @andersy005 described.

ian-r-rose commented 5 years ago

@jbaksta indicated that the root of the problem was with websocket connection settings when using a reverse proxy. That type of thing very quickly gets out of my ability to answer. Can you reproduce the same thing locally or with SSH tunneling?

rabernat commented 5 years ago

It works fine when I use ssh tunneling.

The problem arises when trying to go through nbserverproxy, either directly via the cluster dashboard link in a separate tab or when using the dask-labextension viewers.

ian-r-rose commented 5 years ago

Do I understand you that it also occurs if you open it in a separate browser tab? Do you still have nbserverproxy installed, or are you using the renamed-and-revamped jupyter-server-proxy?

rabernat commented 5 years ago

Do I understand you that it also occurs if you open it in a separate browser tab?

Yes, if I am going via nbserverproxy.

Do you still have nbserverproxy installed, or are you using the renamed-and-revamped jupyter-server-proxy?

I am using nbserverproxy. I had not heard of jupyter-server-proxy.

ian-r-rose commented 5 years ago

Yuvi renamed nbserverproxy to jupyter-server-proxy, and more-or-less rewrote it from the ground up. I'd recommend changing to that and trying again. dask-labextension now depends on jupyter-server-proxy, so I wonder if having both installed might cause some issues.

rabernat commented 5 years ago

Yes that fixed it! Thanks!

FWIW, I did not get jupyter-server-proxy installed when I installed dask-labextension. I had to install it explicitly.

ian-r-rose commented 5 years ago

Great! Glad it's working now!

I'm not sure why jupyter-server-proxy was not installed properly... it's listed in the setup.py: https://github.com/dask/dask-labextension/blob/master/setup.py#L48