search

jupyter / notebook

Jupyter Interactive Notebook
https://jupyter-notebook.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
11.77k stars 5k forks source link

Jupyter can't connect to kernel. #3910

Open abalter opened 6 years ago

abalter commented 6 years ago

I am running jupyter on a digital ocean ubuntu 18.04 VM. I'm using an Apache reverse proxy to map port 8888 to jupyter.myserver.com.

I know my Apache proxy is working because:

  1. I have been running jupyter notebooks successfully until the last few days.
  2. I have other apps (such as rstudio server) I am accessing on different ports with identical reverse proxies.

I know it is not any kind of tcp error on my local machine because:

  1. I have tried turning off all firewalls
  2. I have tried turning off all antivirus programs
  3. I have tried four different browsers, including in safe mode and anonymous windows.
  4. I have tried using FF in my WSL linux installation
  5. I can run a notebook successfully on localhost
  6. I have tried other computers and even my phone.

Symptoms:

  1. I connect to the notebook (either as /tree? or /lab.
  2. I try opening a new terminal. The terminal tab opens, but no prompt appears.
  3. I try opening a notebook and executing something like print("hello"). The cell never executes.
  4. I see the broken link symbol to the right of the kernel's name and the orange "Connecting to kernel" message.
  5. No amount of disconnecting, reconnecting, kernel restarting, opening, closing, trying different kernels, or any other games changes the result.

  6. I get 404 errors in the console where I started the notebook, but no other errors.

    [W 04:56:54.446 NotebookApp] 400 GET /api/kernels/f56f696b-4ccc-4038-9674- 
993d6cbe96be/channels?session_id=e6f4e169ac134aed814f87efc12a23b3 (127.0.0.1) 4.13ms 
referer=None
[W 04:56:55.769 NotebookApp] Replacing stale connection: f56f696b-4ccc-4038-9674- 
993d6cbe96be:e6f4e169ac134aed814f87efc12a23b3
[W 04:57:02.270 NotebookApp] Replacing stale connection: f56f696b-4ccc-4038-9674- 
993d6cbe96be:6766844e2fc0474080895aeb758a5e6b
[W 04:57:55.742 NotebookApp] Replacing stale connection: f56f696b-4ccc-4038-9674- 
993d6cbe96be:6766844e2fc0474080895aeb758a5e6b
  7. There are errors in /var/log/apache2/error.log:
[Thu Sep 13 04:59:12.471533 2018] [proxy_http:error] [pid 8510] (70007)The timeout specified has expired: [client 73.11.33.164:38901] AH01102: error reading status line from remote server localhost:
8888
[Thu Sep 13 04:59:12.471597 2018] [proxy:error] [pid 8510] [client 73.11.33.164:38901] AH00898: Error reading from remote server returned by /api/kernels/f56f696b-4ccc-4038-9674-993d6cbe96be/channel
s
1kastner commented 6 years ago

The communication to the kernel goes through WebSockets. That is not REST! You need to allow the switch of the protocol to WebSockets for the reverse proxy. We did that with nginx so I can't help you with Apache code. The prefix is ws://.

1kastner commented 6 years ago

@abalter could this solve your issue?

Aravinviju commented 5 years ago

I'm getting a smiliar error... #4630

It has something to do with the websocket connection? Can you please explain how to resolve this using nginx? as I have never used or come across nginx before, I don't understand how to sort this out!

Cheers Arav

1kastner commented 5 years ago

@Aravinviju please exactly describe your setup. If you haven't heard of nginx, most likely that is not what you need to configure.

Aravinviju commented 5 years ago

Okay, So I have an GCP instance with GPUs configured in it.

The Instance config is: OS - Ubuntu 16.04 I downloaded Anaconda from http://repo.continuum.io/archive/Anaconda3-2018.12-Linux-x86_64.sh python version 3.6.8

In the jupyter config I add the below for the jupyter to work: c = get_config() c.NotebookApp.ip = '' c.NotebookApp.open_browser = False c.NotebookApp.allow_origin = '' c.NotebookApp.allow_remote_access = True c.NotebookApp.port = 8000

To use the Jupyter notebook in the GCP instance, I SSH instance and initial or start jupyter from the terminal and use the external IP of the GCP instance and the port in which the jupyter is started to open the jupyter tree in the browser. I have been using this for the past 5 months, and haven't face an issue like this before.

From 3rd of may 2019, everything worked good and still works till the jupyter tree opening, but when I start a jupyter notebook I constantly get this error. The error details are in the issue I linked before.

some suggested the tornado downgrading solution, but I'm pretty sure it doesn't work for me as I have tried every version of it.

The Local Machine: Is a windows 7 64 bit machine.

Hope I have given all the config you need. I'm also attaching a copy of the base environment version list i have saved it for future use.

base.txt

Hope this was clear, expecting a proper solution for this as this is holding all the works.

Thanks Arav

1kastner commented 5 years ago

That has nothing to do with my proposed solution, I am sorry for that.

Aravinviju commented 5 years ago

If you have seen the issue I linked to, there is an attachment which shows the websocket issue. Is there and specific solution for that? like something you have specified here: https://github.com/jupyter/notebook/issues/3910#issuecomment-439026820

OR is there any other solution to get rid of the websocket connection failed error with respect to my set up?

jurajbujdoso commented 5 years ago

In my case it was problem with the proxy server. I have add temporary https proxy as frontend and use http jupyter as backend. Then the backend start to work. I assume the solution is depending mostly on your internet access. I get no troubles when I used jupyter notebook without proxy. In my case: https://aptro.github.io/server/architecture/2016/06/21/Jupyter-Notebook-Nginx-Setup.html was not an solution.

st 15. 5. 2019 o 14:23 Aravinviju notifications@github.com napísal(a):

If you have seen the issue I linked to, there is an attachment which shows the websocket issue. Is there and specific solution for that? like something you have specified here: #3910 (comment) https://github.com/jupyter/notebook/issues/3910#issuecomment-439026820

OR is there any other solution to get rid of the websocket connection failed error with respect to my set up?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jupyter/notebook/issues/3910?email_source=notifications&email_token=AH6FXUZEMJQDVBQVVGWS4ETPVP6DNA5CNFSM4FU2J2P2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVOPO7Y#issuecomment-492631935, or mute the thread https://github.com/notifications/unsubscribe-auth/AH6FXU2IVEG2MOMEO6QJFOTPVP6DNANCNFSM4FU2J2PQ .

--

Juraj.Bujdoso@gmail.com

cryptogopher commented 2 years ago

In Apache, besides proxying:

ProxyRequests Off
ProxyPreserveHost On
ProxyPass / http://127.0.0.1:8888/
ProxyPassReverse / http://127.0.0.1:8888/

I had to do some rewriting to get things running correctly:

RewriteEngine on
RewriteCond %{HTTP:Upgrade} =websocket
RewriteRule /(.*) ws://127.0.0.1:8888/$1 [P,L]
RewriteCond %{HTTP:Upgrade} !=websocket
RewriteRule /(.*) http://127.0.0.1:8888/$1 [P,L]

This is assuming your Jupyter server is running on 127.0.0.1:8888.

PPJAISON commented 5 months ago

thanks. it works for me RewriteEngine on RewriteCond %{HTTP:Upgrade} =websocket RewriteRule /(.) ws://127.0.0.1:8888/$1 [P,L] RewriteCond %{HTTP:Upgrade} !=websocket RewriteRule /(.) http://127.0.0.1:8888/$1 [P,L]