jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.04k stars 341 forks source link

Can't connect to the kernel #290

Closed PaluchowskiMatthew closed 5 years ago

PaluchowskiMatthew commented 5 years ago

I successfully installed jupyterhub on a google cloud instance however I am unable to start a working jupyter notebook as it fails to connect to a kernel with a following message:

Connection failed A connection to the notebook server could not be established. The notebook will continue trying to reconnect. Check your network connection or notebook server configuration.

image

jupyterhub.service logs:

root@jupyterhub-0:/opt/tljh/config/jupyterhub_config.d# sudo journalctl -u jupyterhub.service -- Logs begin at Thu 2019-03-07 11:12:13 UTC, end at Thu 2019-03-07 12:07:43 UTC. -- Mar 07 11:16:46 jupyterhub-0 systemd[1]: /etc/systemd/system/jupyterhub.service:12: Failed to parse protect home value, ignoring: tmpfs Mar 07 11:16:46 jupyterhub-0 systemd[1]: Started jupyterhub.service. Mar 07 11:16:46 jupyterhub-0 systemd[1]: Stopping jupyterhub.service... Mar 07 11:16:46 jupyterhub-0 systemd[1]: Stopped jupyterhub.service. Mar 07 11:16:46 jupyterhub-0 systemd[1]: Started jupyterhub.service. Mar 07 11:16:46 jupyterhub-0 systemd[1]: /etc/systemd/system/jupyterhub.service:12: Failed to parse protect home value, ignoring: tmpfs Mar 07 11:16:47 jupyterhub-0 systemd[1]: /etc/systemd/system/jupyterhub.service:12: Failed to parse protect home value, ignoring: tmpfs Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.206 JupyterHub app:1673] Using Authenticator: firstuseauthenticator.firstuseauthenticator.FirstUseAuthenticator Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.207 JupyterHub app:1673] Using Spawner: builtins.UserCreatingSpawner Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.209 JupyterHub app:1055] Writing cookie_secret to /opt/tljh/state/jupyterhub_cookie_secret Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.274 alembic.runtime.migration migration:130] Context impl SQLiteImpl. Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.276 alembic.runtime.migration migration:137] Will assume non-transactional DDL. Mar 07 11:16:48 jupyterhub-0 python3[7815]: [I 2019-03-07 11:16:48.289 alembic.runtime.migration migration:356] Running stamp_revision -> 896818069c98 Mar 07 11:17:33 jupyterhub-0 useradd[7938]: new group: name=jupyter-deeption, GID=1014 Mar 07 11:17:33 jupyterhub-0 useradd[7938]: new user: name=jupyter-deeption, UID=1011, GID=1014, home=/home/jupyter-deeption, shell=/bin/sh Mar 07 11:17:33 jupyterhub-0 python3[7815]: Adding user jupyter-deeption to group jupyterhub-users Mar 07 11:17:33 jupyterhub-0 gpasswd[7945]: user jupyter-deeption added by root to group jupyterhub-users Mar 07 11:17:33 jupyterhub-0 python3[7815]: Adding user jupyter-deeption to group jupyterhub-admins Mar 07 11:17:33 jupyterhub-0 gpasswd[7949]: user jupyter-deeption added by root to group jupyterhub-admins Mar 07 11:17:33 jupyterhub-0 python3[7815]: Running as unit: jupyter-deeption.service

jupyter-deeption (user) logs:

root@jupyterhub-0:/opt/tljh/config/jupyterhub_config.d# sudo journalctl -u jupyter-deeption -- Logs begin at Thu 2019-03-07 11:12:13 UTC, end at Thu 2019-03-07 12:03:15 UTC. -- Mar 07 11:17:33 jupyterhub-0 systemd[1]: Started /bin/bash -c cd /home/jupyter-deeption && exec jupyterhub-singleuser --port=54437. Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.639 SingleUserNotebookApp extension:168] JupyterLab extension loaded from /opt/tljh/user/lib/python3.6/site-packages/jupyterlab Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.641 SingleUserNotebookApp extension:169] JupyterLab application directory is /opt/tljh/user/share/jupyter/lab Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.677 SingleUserNotebookApp extension:23] nteract extension loaded from /opt/tljh/user/lib/python3.6/site-packages/nteract_on_jupyter Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.679 SingleUserNotebookApp singleuser:406] Starting jupyterhub-singleuser server version 0.9.4 Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.688 SingleUserNotebookApp notebookapp:1712] Serving notebooks from local directory: /home/jupyter-deeption Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.688 SingleUserNotebookApp notebookapp:1712] The Jupyter Notebook is running at: Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.689 SingleUserNotebookApp notebookapp:1712] http://127.0.0.1:54437/user/deeption/ Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.689 SingleUserNotebookApp notebookapp:1713] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). Mar 07 11:17:34 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:34.795 SingleUserNotebookApp log:158] 302 GET /user/deeption/ -> /user/deeption/tree? (@127.0.0.1) 9.27ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.122 SingleUserNotebookApp log:158] 302 GET /user/deeption/?redirects=1 -> /user/deeption/tree?redirects=1 (@185.127.206.173) 1.22ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.149 SingleUserNotebookApp log:158] 302 GET /user/deeption/tree?redirects=1 -> /hub/api/oauth2/authorize?client_id=jupyterhub-user-deeption&redirect_uri=%2Fuser%2Fdeeption%2Foauth_callback&response_type=code&state=[secret] (@185.127.206.173) 3.46m Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.318 SingleUserNotebookApp auth:875] Logged-in user {'kind': 'user', 'name': 'deeption', 'admin': True, 'groups': [], 'server': '/user/deeption/', 'pending': None, 'created': '2019-03-07T11:16:48.410639Z', 'last_activity': '2019-03-07T11:17:35.309 Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.320 SingleUserNotebookApp log:158] 302 GET /user/deeption/oauthcallback?code=[secret]&state=[secret] -> /user/deeption/tree?redirects=1 (@185.127.206.173) 80.10ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.404 SingleUserNotebookApp log:158] 200 GET /user/deeption/tree?redirects=1 (deeption@185.127.206.173) 58.36ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.922 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/tree?=1551957455858 (deeption@185.127.206.173) 3.65ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.928 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957455860 (deeption@185.127.206.173) 3.62ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.931 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/common?=1551957455859 (deeption@185.127.206.173) 5.10ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.938 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/kernelspecs (deeption@185.127.206.173) 5.39ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.940 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/sessions?=1551957455861 (deeption@185.127.206.173) 6.59ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.942 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957455862 (deeption@185.127.206.173) 1.34ms Mar 07 11:17:35 jupyterhub-0 bash[7955]: [I 2019-03-07 11:17:35.998 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents?type=directory&=1551957455863 (deeption@185.127.206.173) 3.94ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.523 SingleUserNotebookApp log:158] 302 GET /user/deeption/ -> /user/deeption/tree? (@185.127.206.173) 1.40ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.547 SingleUserNotebookApp log:158] 200 GET /user/deeption/tree (deeption@185.127.206.173) 3.15ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.653 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/tree?=1551957522610 (deeption@185.127.206.173) 4.31ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.656 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/common?=1551957522611 (deeption@185.127.206.173) 6.67ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.678 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957522612 (deeption@185.127.206.173) 5.24ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.681 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/sessions?=1551957522613 (deeption@185.127.206.173) 2.31ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.684 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/terminals?=1551957522614 (deeption@185.127.206.173) 1.54ms Mar 07 11:18:42 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:42.703 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents?type=directory&_=1551957522615 (deeption@185.127.206.173) 2.71ms Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.049 SingleUserNotebookApp handlers:153] Creating new notebook in Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.071 SingleUserNotebookApp sign:396] Writing notebook-signing key to /home/jupyter-deeption/.local/share/jupyter/notebook_secret Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.083 SingleUserNotebookApp log:158] 201 POST /user/deeption/api/contents (deeption@185.127.206.173) 34.74ms Mar 07 11:18:47 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:47.371 SingleUserNotebookApp log:158] 200 GET /user/deeption/notebooks/Untitled.ipynb?kernelname=python3 (deeption@185.127.206.173) 261.87ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.093 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/notebook?=1551957527975 (deeption@185.127.206.173) 6.34ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.098 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/config/common?=1551957527976 (deeption@185.127.206.173) 10.02ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.157 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents/Untitled.ipynb?type=notebook&=1551957527977 (deeption@185.127.206.173) 7.11ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.274 SingleUserNotebookApp kernelmanager:164] Kernel started: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969 Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.281 SingleUserNotebookApp log:158] 201 POST /user/deeption/api/sessions (deeption@185.127.206.173) 37.64ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.295 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents/Untitled.ipynb/checkpoints?_=1551957527978 (deeption@185.127.206.173) 5.95ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.346 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 8.49ms Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.905 SingleUserNotebookApp handlers:190] Adapting to protocol v5.1 for kernel 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969 Mar 07 11:18:48 jupyterhub-0 bash[7955]: /opt/tljh/user/lib/python3.6/site-packages/notebook/base/zmqhandlers.py:284: RuntimeWarning: coroutine 'WebSocketHandler.get' was never awaited Mar 07 11:18:48 jupyterhub-0 bash[7955]: super(AuthenticatedZMQStreamHandler, self).get(*args, **kwargs) Mar 07 11:18:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:48.908 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/kernels/4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969/channels?sessionid=99809f1975b44be79ec70788f2ee4b01 (deeption@185.127.206.173) 580.28ms Mar 07 11:18:50 jupyterhub-0 bash[7955]: [W 2019-03-07 11:18:50.183 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01 Mar 07 11:18:53 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:53.337 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.43ms Mar 07 11:18:58 jupyterhub-0 bash[7955]: [I 2019-03-07 11:18:58.336 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.58ms Mar 07 11:19:37 jupyterhub-0 bash[7955]: [W 2019-03-07 11:19:37.266 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01 Mar 07 11:19:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:19:48.347 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.56ms Mar 07 11:19:53 jupyterhub-0 bash[7955]: [I 2019-03-07 11:19:53.346 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.56ms Mar 07 11:20:26 jupyterhub-0 bash[7955]: [W 2019-03-07 11:20:26.303 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01 Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.259 SingleUserNotebookApp log:158] 200 GET /user/deeption/api/contents/Untitled.ipynb?content=0&=1551957527979 (deeption@185.127.206.173) 4.81ms Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.283 SingleUserNotebookApp handlers:164] Saving file at /Untitled.ipynb Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.297 SingleUserNotebookApp log:158] 200 PUT /user/deeption/api/contents/Untitled.ipynb (deeption@185.127.206.173) 15.73ms Mar 07 11:20:48 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:48.351 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.84ms Mar 07 11:20:53 jupyterhub-0 bash[7955]: [I 2019-03-07 11:20:53.352 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 5.50ms Mar 07 11:21:19 jupyterhub-0 bash[7955]: [W 2019-03-07 11:21:19.410 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01 Mar 07 11:21:23 jupyterhub-0 bash[7955]: [I 2019-03-07 11:21:23.358 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 4.60ms Mar 07 11:21:28 jupyterhub-0 bash[7955]: [I 2019-03-07 11:21:28.358 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 5.06ms Mar 07 11:22:20 jupyterhub-0 bash[7955]: [W 2019-03-07 11:22:20.556 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01 Mar 07 11:22:38 jupyterhub-0 bash[7955]: [I 2019-03-07 11:22:38.399 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 38.32ms Mar 07 11:22:43 jupyterhub-0 bash[7955]: [I 2019-03-07 11:22:43.403 SingleUserNotebookApp log:158] 200 GET /user/deeption/metrics (deeption@185.127.206.173) 5.15ms Mar 07 11:23:37 jupyterhub-0 bash[7955]: [W 2019-03-07 11:23:37.682 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01 Mar 07 11:25:26 jupyterhub-0 bash[7955]: [W 2019-03-07 11:25:26.791 SingleUserNotebookApp handlers:254] Replacing stale connection: 4dfdd4e6-aaeb-44c1-97dc-5b7fa5cac969:99809f1975b44be79ec70788f2ee4b01

I tried downgrading tornado as suggested here https://github.com/jupyter/notebook/issues/4399 however it did not resolve my issue... Any suggestions?

yuyueugene84 commented 5 years ago

I can confirm that the same issue also occurs when I installed Jupyterhub in AWS.

PaluchowskiMatthew commented 5 years ago

Apparently downgrading the tornado was the right call it just needs to be done in a right way: From within running jupyterhub login as an admin user, open new terminal window and run

sudo -E pip install tornado==5.1.1

to downgrade tornado.

Then make sure to restart the server (within jupyrehub click on the control panel in the upper right corner, stop the server then start it again. From now on you should be able to connect to the notebook)

willirath commented 5 years ago

Let's keep open until fixed.

willirath commented 5 years ago

@PaluchowskiMatthew Do you have a chance to check #292?

Would be done as follows:

export \ TLJH_BOOTSTRAP_PIP_SPEC="git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@${branch_name}"

curl \ https://raw.githubusercontent.com/jupyterhub/the-littlest-jupyterhub/${branch_name}/bootstrap/bootstrap.py \ | sudo -E python3 - --admin

PaluchowskiMatthew commented 5 years ago

@willirath Problem persists and tornado version remains unchanged (6.0.1)

image

yuvipanda commented 5 years ago

Should hopefully be fixed now with https://github.com/jupyterhub/the-littlest-jupyterhub/pull/292. Please re-open if that isn't the case!

awesome-crab commented 1 year ago

still experience this issue. downgrading tornado is not so easy since there are about 10 packages relying on the latest version of tornado. any help would be appreciated