Nginx + notebook issue

[jsalva]

Hi guys,

I've been trying to get my jupyter notebook to run behind an nginx reverse proxy and everything seems to be going well, until i actually try to connect to the kernel in a notebook. Here's my /etc/nginx/conf.d/configuration.conf configuration upfront:

upstream notebook {
    server localhost:8888;
}

server {
    listen                    443;
    ssl                       on;
    server_name               notebook.mydomain.com www.notebook.mydomain.com;
    ssl_certificate           /home/<user>/certificates/ipythoncert.pem;
    ssl_certificate_key       /home/<user>/certificates/ipythoncert.pem;
    ssl_protocols             TLSv1 TLSv1.1 TLSv1.2;
    access_log                /home/<user>/.jupyter/log/nginx.access.log;
    error_log                 /home/<user>/.jupyter/log/nginx.error.log;

    location / {
        proxy_pass            https://notebook;
        proxy_set_header      Host $host;
    }

    location /api/kernels/ {
        proxy_pass            https://notebook;
        proxy_set_header      Host $host;
        # websocket support
        proxy_http_version    1.1;
        proxy_set_header      Upgrade "websocket";
        proxy_set_header      Connection "Upgrade";
        proxy_read_timeout    86400;
    }
}

Also, here's my ~/.jupyter/jupyter_notebook_config.py:

c.NotebookApp.ip = 'localhost'
c.NotebookApp.allow_origin = '*'
c.NotebookApp.certfile = u'/home/<user>/certificates/ipythoncert.pem'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:imnotthatstupiddontworry'
c.NotebookApp.trust_xheaders = True
#everything else is left commented

On the client, the <div id="notification_kernel">...</div> won't stop displaying "connecting to kernel", and I can't get my cells to run.

As it retries, I get a popup with "A connection to the notebook server could not be established. The notebook will continue trying to reconnect, but until it does, you will NOT be able to run code. Check your network connection or notebook server configuration."

and, the javascript console:

> Default extension for cell metadata editing loaded.
> Raw Cell Format toolbar preset loaded.
> Slideshow extension for metadata editing loaded.
> https://notebook.mydomain.com/nbextensions/widgets/notebook/js/extension.js Failed to load resource: the server responded with a status of 404 (Not Found)
> ipywidgets package not installed.  Widgets are not available.
> Session: kernel_created (5e294362-a965-46c0-9bef-cfd052f91f9e)
> Starting WebSockets: wss://notebook.mydomain.com/api/kernels/a7fdec59-190e-405b-bd2f-35126d0b8981
> WebSocket connection to 'wss://notebook.mydomain.com/api/kernels/a7fdec59-190e-405b-bd2f-35126d0b8981/channels?session_id=<session_id>' failed: Error during  WebSocket handshake: Unexpected response code: 504
> Kernel: kernel_disconnected (a7fdec59-190e-405b-bd2f-35126d0b8981)
> WebSocket connection failed:  wss://notebook.mydomain.com/api/kernels/<kernel_id>
> Connection lost, reconnecting in 1 seconds.
> Kernel: kernel_reconnecting (a7fdec59-190e-405b-bd2f-35126d0b8981)
...
more failures
...

(Edit: I just noticed that in the above javascript console output, the id following the kernel_created message is different from all the other ids used in e.g. the kernel URLs; my guess is that isn't relevant at all, but I'm running out of ideas...)

from the server's perspective (loglevel "DEBUG"):

[D 21:48:20.451 NotebookApp] Using contents: services/contents
[D 21:48:20.452 NotebookApp] 200 GET /notebooks/Odd%20jobs.ipynb (<my-IP-address>) 1.86ms
[D 21:48:20.574 NotebookApp] 200 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=9b2c8d3489227115310662a343fce11c (<my-IP-address>) 0.82ms
[D 21:48:20.583 NotebookApp] 200 GET /static/components/MathJax/MathJax.js?config=TeX-AMS_HTML-full,Safe&delayStartupUntil=configured (<my-IP-address>) 0.77ms
[D 21:48:20.588 NotebookApp] 200 GET /static/components/bootstrap-tour/build/css/bootstrap-tour.min.css?v=d0b3c2fce6056a2ddd5a4513762a94c4 (<my-IP-address>) 0.69ms
[D 21:48:20.594 NotebookApp] 200 GET /static/components/codemirror/lib/codemirror.css?v=549e7432c1e8e94d1a2e66d7be92a430 (<my-IP-address>) 0.63ms
[D 21:48:20.596 NotebookApp] 200 GET /static/style/style.min.css?v=726bda814675157da88304dabb766757 (<my-IP-address>) 1.27ms
[D 21:48:20.610 NotebookApp] 200 GET /static/notebook/css/override.css?v=e6f18013b8771987812e992b38ec3318 (<my-IP-address>) 0.75ms
[D 21:48:20.622 NotebookApp] 200 GET /custom/custom.css (<my-IP-address>) 0.75ms
[D 21:48:20.635 NotebookApp] 200 GET /static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe (<my-IP-address>) 0.55ms
[D 21:48:20.637 NotebookApp] 200 GET /static/components/requirejs/require.js?v=2de44fdcc1fe5e939aa4ce80626b241d (<my-IP-address>) 0.78ms
[D 21:48:20.651 NotebookApp] 200 GET /static/components/text-encoding/lib/encoding.js?v=d5bb0fc9ffeff7d98a69bb83daa51052 (<my-IP-address>) 0.87ms
[D 21:48:20.665 NotebookApp] 200 GET /static/notebook/js/main.min.js?v=40e10638fcf65fc1c057bff31d165e9d (<my-IP-address>) 5.20ms
[D 21:48:20.680 NotebookApp] 200 GET /static/base/images/logo.png?v=7c4597ba713d804995e8f8dad448a397 (<my-IP-address>) 0.70ms
[D 21:48:20.784 NotebookApp] 200 GET /static/components/MathJax/config/TeX-AMS_HTML-full.js?rev=2.5.3 (<my-IP-address>) 1.57ms
[D 21:48:20.914 NotebookApp] 200 GET /static/style/style.min.css.map (<my-IP-address>) 1.13ms
[D 21:48:21.387 NotebookApp] 200 GET /static/components/MathJax/config/Safe.js?rev=2.5.3 (<my-IP-address>) 0.68ms
[D 21:48:21.877 NotebookApp] 200 GET /static/notebook/js/main.min.js.map (<my-IP-address>) 5.87ms
[D 21:48:22.066 NotebookApp] 200 GET /static/services/contents.js?v=20151020214716 (<my-IP-address>) 1.13ms
[D 21:48:22.080 NotebookApp] 200 GET /custom/custom.js?v=20151020214716 (<my-IP-address>) 0.74ms
[D 21:48:22.132 NotebookApp] 200 GET /api/config/notebook?_=1445377700749 (<my-IP-address>) 0.74ms
[D 21:48:22.135 NotebookApp] 200 GET /api/config/common?_=1445377700750 (<my-IP-address>) 0.54ms
[D 21:48:22.491 NotebookApp] Native kernel (python2) available from /home/<user>/.virtualenvs/<env>/lib/python2.7/site-packages/ipykernel/resources
[D 21:48:22.491 NotebookApp] Native kernel (python2) available from /home/<user>/.virtualenvs/<env>/lib/python2.7/site-packages/ipykernel/resources
[D 21:48:22.491 NotebookApp] 200 GET /api/kernelspecs (<my-IP-address>) 1.36ms
[D 21:48:22.496 NotebookApp] 200 GET /static/components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 (<my-IP-address>) 0.82ms
[D 21:48:22.541 NotebookApp] 200 GET /api/contents/Odd%20jobs.ipynb?type=notebook&_=1445377700751 (<my-IP-address>) 9.91ms
[D 21:48:22.671 NotebookApp] Using contents: services/contents
[W 21:48:22.672 NotebookApp] 404 GET /nbextensions/widgets/notebook/js/extension.js?v=20151020214716 (<my-IP-address>) 1.60ms referer=https://notebook.mydomain.com/notebooks/Odd%20jobs.ipynb
[D 21:48:22.729 NotebookApp] 200 GET /static/components/MathJax/jax/output/HTML-CSS/fonts/STIX-Web/fontdata.js?rev=2.5.3 (<my-IP-address>) 0.72ms
[D 21:48:23.067 NotebookApp] 201 POST /api/sessions (<my-IP-address>) 0.86ms
[D 21:48:23.070 NotebookApp] 200 GET /api/contents/Odd%20jobs.ipynb/checkpoints?_=1445377700752 (<my-IP-address>) 0.70ms
[D 21:48:23.089 NotebookApp] Native kernel (python2) available from /home/<user>/.virtualenvs/<env>/lib/python2.7/site-packages/ipykernel/resources
[D 21:48:23.089 NotebookApp] Serving kernel resource from: /home/<user>/.virtualenvs/<env>/lib/python2.7/site-packages/ipykernel/resources
[D 21:48:23.090 NotebookApp] 200 GET /kernelspecs/python2/logo-64x64.png (<my-IP-address>) 1.45ms
[D 21:48:23.165 NotebookApp] 200 GET /static/components/MathJax/extensions/Safe.js?rev=2.5.3 (<my-IP-address>) 0.68ms
[D 21:48:23.360 NotebookApp] Initializing websocket connection /api/kernels/5bffc82f-1765-4fc4-b3cc-3a634ac571d9/channels
[D 21:48:23.363 NotebookApp] Opening websocket /api/kernels/5bffc82f-1765-4fc4-b3cc-3a634ac571d9/channels
[D 21:48:23.363 NotebookApp] Connecting to: tcp://127.0.0.1:47444
[D 21:48:23.364 NotebookApp] Connecting to: tcp://127.0.0.1:36129
[D 21:48:23.364 NotebookApp] Connecting to: tcp://127.0.0.1:53437

The only way I got the websockets to connect to the kernel properly was to bypass nginx and set

c.NotebookApp.ip = '*'

Then, when I access it from the https://ec2-url-for-my-instance:8888 (after clicking through the security warnings that pop up with my self-signed certificate), the websockets will connect to the kernel and I can go on my merry way running whatever I want...

I have been trying to determine why the nginx proxy isn't working for almost 2 days now.

Does anyone have any ideas about what's going on?

[minrk]

@rgbkrk ideas? @jsalva have you tried switch from 'localhost' to '127.0.0.1' explicitly?

[jsalva]

I just tried updating every occurrence of localhost to 127.0.0.1 in ~/.jupyter/jupyter_notebook_config.py, as well as in /etc/nginx/conf.d/configuration.conf, to no avail

[ianabc]

@jsalva did you get anywhere with this? I'm doing something similar with apache and running into the same problem.

[jsalva]

I wish I could have some helpful updates, but nope, gave up and moved on.

[markoverholser]

I run mine behind a reverse proxy on Apache, and it took me a full evening of googling and editing my configuration and restarting, but I was able to get it with these directives:

<Location /notebook>
    RequestHeader unset Accept-Encoding
    ProxyPass        http://notebook:8888/notebook
    ProxyPassReverse http://notebook:8888/notebook
    ProxyPreserveHost on
    Order allow,deny
    Allow from all
</Location>
<Location /notebook/api/kernels/>
    ProxyPass        ws://notebook:8888/notebook/api/kernels/
    ProxyPassReverse ws://notebook:8888/notebook/api/kernels/
</Location>

Had to enable some additional modules for Apache for this to work (on Ubuntu 14.04 LTS):

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod proxy_https
sudo a2enmod proxy_wstunnel
sudo a2enmod headers

Also, in my .jupyter/jupyter_notebook_config.py I changed the base URL for the notebook (since it's being hosted as a virtual folder this makes the reverse proxying easier):

c.NotebookApp.base_url = '/notebook'

@ianabc @jsalva Feel free to ping me for discussion. I am not 100% sure what your situations are, or if this helps, and I am hardly an Apache guy, but I did manage to make it work.

[ianabc]

@hosemn I was missing the proxy_wstunnel module, when turned on debugging I could see the protocol error. Thanks for your help!

[markoverholser]

Glad it worked for you, and I hope it helps other Apache users. As for nginx, I am not sure where to start.

[cboettig]

Hmm, I also cannot get nginx proxy or a caddyserver proxy working. I can get pretty close, can log in and create and delete files, but cannot connect to a python kernel (or launch a terminal). Must have something to do with the websockets proxy, but it's beyond me. The Jupyter logs just show 400 errors like: [W 01:51:26.202 NotebookApp] 400 GET /api/kernels/aa45474d-7fb8-4685-98dc-5ebeba378f41/channels?session_id=1A9BFB019EE0472486650C81705BCC73 (172.17.0.1) 5.77ms referer=None

My nginx conf is here: https://gist.github.com/cboettig/8643341bd3c93b62b5c2 (simple variations, e.g. serving from root instead of a subdomain, or serving without the ssl parts still give me the same issue). Any suggestions?

[ellisonbg]

Don't know if it will help, but here is a working nginx config file for jupyterhub:

https://github.com/calpolydatascience/jupyterhub-deploy-data301/blob/master/roles/nginx/templates/nginx.conf.j2

On Wed, Dec 30, 2015 at 6:01 PM, Carl Boettiger notifications@github.com wrote:

Hmm, I also cannot get nginx proxy or a caddyserver proxy working. I can get pretty close, can log in and create and delete files, but cannot connect to a python kernel (or launch a terminal). Must have something to do with the websockets proxy, but it's beyond me. The Jupyter logs just show 400 errors like: [W 01:51:26.202 NotebookApp] 400 GET /api/kernels/aa45474d-7fb8-4685-98dc-5ebeba378f41/channels?session_id=1A9BFB019EE0472486650C81705BCC73 (172.17.0.1) 5.77ms referer=None

My nginx conf is here: https://gist.github.com/cboettig/8643341bd3c93b62b5c2 (simple variations, e.g. serving from root instead of a subdomain, or serving without the ssl parts still give me the same issue). Any suggestions?

— Reply to this email directly or view it on GitHub https://github.com/jupyter/notebook/issues/625#issuecomment-168110162.

Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub bgranger@calpoly.edu and ellisonbg@gmail.com

[cboettig]

@ellisonbg Thanks for the suggestion, I've actually tried that config as well (after putting in the appropriate values for the ansible-template chunks), and I get the exact same issue -- nginx shows 400 errors trying to connect to the kernel, and Jupyter opens but cannot reach a python kernel...

I'm running the jupyter instance via a docker container from jupyter/docker-stacks; but don't see why that should be messing up the connection to the kernel when behind a proxy...

[rcompton]

@cboettig Exact same issue here (using nginx in front of https://hub.docker.com/r/jupyter/jupyterhub/~/dockerfile/ ). No idea what could be wrong.

nginx tells me "client closed connection while waiting for request"

Update: Turns out my issue was that my nginx server was behind an aws ELB (for https). ELBs need special configs for websockets. I ended up just bypassing the ELB.

[cboettig]

@rcompton Thanks, turned out my issue was also websockets related (and had hub-style redirects with user bit in my previous file). Here's the tweaked nginx working for me now: https://gist.github.com/cboettig/8643341bd3c93b62b5c2

[nicerobot]

It works for me using jsalva's config with one small change:

location ~ /api/kernels/ {

[AlfioEmanueleFresta]

@nicerobot's suggestion worked for me. Thanks!

[qqrs]

This worked for me, based on jsalva's post (behind VPN so no SSL):

server {
    listen 80 default_server;

    location / {
        proxy_pass            http://localhost:{{ jupyter_server_port }};
        proxy_set_header      Host $host;
    }

    location /api/kernels/ {
        proxy_pass            http://localhost:{{ jupyter_server_port }};
        proxy_set_header      Host $host;
        # websocket support
        proxy_http_version    1.1;
        proxy_set_header      Upgrade "websocket";
        proxy_set_header      Connection "Upgrade";
        proxy_read_timeout    86400;
    }
}

[wintermelons]

Just ran into this issue, and tried the solution by @qqrs, but it isn't exactly future-proof and some functionalities like spawning terminals still didn't work. I dug a little deeper and found that websockets in Nginx only needs a specified http version, and being able to upgrade the connection into a tunnel. I currently have this config working:

map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      close;
}

server {
        listen 80;
        server_name jupyter.somewhere.cool;

        location / {
                proxy_pass            http://localhost:9999;
                proxy_set_header      Host $host;
                proxy_http_version    1.1;
                proxy_set_header      Upgrade $http_upgrade;
                proxy_set_header      Connection $connection_upgrade;
        }
}

See http://nginx.org/en/docs/http/websocket.html

[phariel]

@wintermelons it's awesome, you saved my day!

[csymeonides]

I'm still getting the same error even with the solution by @qqrs above. I'm running the notebook server in gcloud with https and OAuth2. Has anyone managed to get it to work in a similar setup?

[nikAizuddin]

Using proxy_set_header Host $host; doesn't work for me. I got "API request failed (403): Forbidden" when I clicked "Stop My Server" button. Error from Jupyterhub log: 403 DELETE /hub/api/users/tester/server (@10.10.102.28)

Changing to proxy_set_header Host $http_host; does solve my issue. So now when I clicked the button, Jupyterhub produce log output with: 204 DELETE /hub/api/users/tester/server (tester@10.10.102.28)

[reichardtj]

I'm running Jupyter behind an ssl reverse proxy. @wintermelons solution did work for me, when accessing the notebook from Firefox - but not from Safari where I cannot connect to the notebook kernel and get a 401 error on the web socket call:

[Error] WebSocket connection to 'wss://somedomain.com/notebooks/api/kernels/6343900b-12f3-4245-81cd-8c1e3642d6b5/channels?session_id=DF7A59E736A041458B79DD76D45AC89B' failed: Unexpected response code: 401

Maybe that does explain the different results people are reporting here.

[372046933]

I have to set proxy_read_timeout to a large value like 86400, otherwise, browser console continues reconnecting. https://github.com/jupyterhub/jupyterhub/issues/3368

[372046933]

I have to set proxy_read_timeout to a large value like 86400, otherwise, browser console continues reconnecting. jupyterhub/jupyterhub#3368

proxy_read_timeout = 60s is enough. Because the ping-pong interval is 30s

[murleo]

For me proxy_set_header Origin ""; solved the problem (SSL on). Reference (in the comments): https://stackoverflow.com/a/23912400

[jojupiter]

Thanks for your code. I advise you to check also if the configuration file (a python file) has no coding error

[LaurentiuAndrei]

Thanks to whoever mentioned web sockets. I am using nginx proxy manager for reverse proxy and all I had to do was tick the "websockets support" to ON for jupyter. This fixed the "400 GET" error logs towards api/kernels in the docker container for jupyer.

[Gakhramanzode]

this helped me

ingress:
  enabled: true
  annotations:
    nginx.org/websocket-services: "proxy-public"
  hosts:
    - example.com

I used helm chart