jupyterhub / systemdspawner

Spawn JupyterHub single-user notebook servers with systemd
BSD 3-Clause "New" or "Revised" License
92 stars 45 forks source link

Unable to respawn after restarting JupyterHub #142

Open ryland-goldman opened 2 months ago

ryland-goldman commented 2 months ago

Bug description

Whenever I use systemctl restart jupyterhub.service, I am unable to spawn a new server session. I get this message: Screenshot 2024-08-24 at 12 30 14

I am able to fix it by manually removing the file /run/jupyter-[username]-singleuser but I feel like there's something wrong with my configuration.

Your personal set up

Logs ``` Aug 24 19:20:25 myserver jupyterhub[790246]: [I 2024-08-24 19:20:25.712 JupyterHub proxy:752] Starting proxy @ http://127.0.0.1:8193 Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.259 [ConfigProxy] info: Proxying http://127.0.0.1:8193 to (no default) Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.267 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.418 JupyterHub app:3690] Hub API listening on http://127.0.0.1:8081/hub/ Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.418 [ConfigProxy] info: 200 GET /api/routes Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.421 [ConfigProxy] info: 200 GET /api/routes Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.421 JupyterHub proxy:477] Adding route for Hub: / => http://127.0.0.1:8081 Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.424 [ConfigProxy] info: Adding route / -> http://127.0.0.1:8081 Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.426 [ConfigProxy] info: Route added / -> http://127.0.0.1:8081 Aug 24 19:20:26 myserver jupyterhub[790249]: 19:20:26.427 [ConfigProxy] info: 201 POST /api/routes/ Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.427 JupyterHub app:3731] JupyterHub is now running at http://127.0.0.1:8193 Aug 24 19:20:26 myserver jupyterhub[790246]: [I 2024-08-24 19:20:26.807 JupyterHub provider:661] Creating oauth client jupyterhub-user-[username] Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.842 JupyterHub user:1007] Unhandled error starting [username]'s server: [Errno 17] File exists: '/run/jupyter-[username]-singleuser' Aug 24 19:20:26 myserver jupyterhub[790246]: Traceback (most recent call last): Aug 24 19:20:26 myserver jupyterhub[790246]: File "/[venv directory]/lib/python3.12/site-packages/jupyterhub/user.py", line 920, in spawn Aug 24 19:20:26 myserver jupyterhub[790246]: await asyncio.wait_for(f, timeout=spawner.start_timeout) Aug 24 19:20:26 myserver jupyterhub[790246]: File "/usr/lib/python3.12/asyncio/tasks.py", line 520, in wait_for Aug 24 19:20:26 myserver jupyterhub[790246]: return await fut Aug 24 19:20:26 myserver jupyterhub[790246]: ^^^^^^^^^ Aug 24 19:20:26 myserver jupyterhub[790246]: File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemdspawner.py", line 331, in start Aug 24 19:20:26 myserver jupyterhub[790246]: await systemd.start_transient_service( Aug 24 19:20:26 myserver jupyterhub[790246]: File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemd.py", line 143, in start_transient_service Aug 24 19:20:26 myserver jupyterhub[790246]: environment_file = make_environment_file( Aug 24 19:20:26 myserver jupyterhub[790246]: ^^^^^^^^^^^^^^^^^^^^^^ Aug 24 19:20:26 myserver jupyterhub[790246]: File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemd.py", line 54, in make_environment_file Aug 24 19:20:26 myserver jupyterhub[790246]: ensure_environment_directory(environment_file_directory) Aug 24 19:20:26 myserver jupyterhub[790246]: File "/[venv directory]/lib/python3.12/site-packages/systemdspawner/systemd.py", line 26, in ensure_environment_directory Aug 24 19:20:26 myserver jupyterhub[790246]: os.makedirs(environment_file_directory, mode=0o700, exist_ok=True) Aug 24 19:20:26 myserver jupyterhub[790246]: File "", line 225, in makedirs Aug 24 19:20:26 myserver jupyterhub[790246]: FileExistsError: [Errno 17] File exists: '/run/jupyter-[username]-singleuser' Aug 24 19:20:26 myserver jupyterhub[790246]: Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.884 JupyterHub pages:312] Error starting server [username]: [Errno 17] File exists: '/run/jupyter-[username]-singleuser' Aug 24 19:20:26 myserver jupyterhub[790246]: Traceback (most recent call last): Aug 24 19:20:26 myserver jupyterhub[790246]: NoneType: None Aug 24 19:20:26 myserver jupyterhub[790246]: Aug 24 19:20:26 myserver jupyterhub[790246]: [W 2024-08-24 19:20:26.884 JupyterHub web:1873] 500 GET /hub/spawn (127.0.0.1): Unhandled error starting server [username] Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.922 JupyterHub log:184] { Aug 24 19:20:26 myserver jupyterhub[790246]: "X-Forwarded-Port": "80", Aug 24 19:20:26 myserver jupyterhub[790246]: "Connection": "Keep-Alive", Aug 24 19:20:26 myserver jupyterhub[790246]: "X-Forwarded-Server": "[domain]", Aug 24 19:20:26 myserver jupyterhub[790246]: "X-Forwarded-Host": "[domain]", Aug 24 19:20:26 myserver jupyterhub[790246]: "X-Forwarded-For": "[ipaddress],127.0.0.1", Aug 24 19:20:26 myserver jupyterhub[790246]: "X-Forwarded-Proto": "https,http", Aug 24 19:20:26 myserver jupyterhub[790246]: "Accept-Language": "en-GB,en;q=0.9", Aug 24 19:20:26 myserver jupyterhub[790246]: "Sec-Fetch-Dest": "document", Aug 24 19:20:26 myserver jupyterhub[790246]: "Referer": "https://[domain]/user/[username]/lab/tree/src/py/refinitiv/Notebooks/Untitled.ipynb", Aug 24 19:20:26 myserver jupyterhub[790246]: "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15", Aug 24 19:20:26 myserver jupyterhub[790246]: "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", Aug 24 19:20:26 myserver jupyterhub[790246]: "Sec-Fetch-Mode": "navigate", Aug 24 19:20:26 myserver jupyterhub[790246]: "Cookie": "_xsrf=[secret]; jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]", Aug 24 19:20:26 myserver jupyterhub[790246]: "Accept-Encoding": "gzip, deflate, br", Aug 24 19:20:26 myserver jupyterhub[790246]: "Sec-Fetch-Site": "same-origin", Aug 24 19:20:26 myserver jupyterhub[790246]: "Host": "[domain]" Aug 24 19:20:26 myserver jupyterhub[790246]: } Aug 24 19:20:26 myserver jupyterhub[790246]: [E 2024-08-24 19:20:26.923 JupyterHub log:192] 500 GET /hub/spawn ([username]@127.0.0.1) 161.32ms Aug 24 19:20:27 myserver jupyterhub[790246]: [I 2024-08-24 19:20:27.154 JupyterHub log:192] 302 GET /user/[username]/api/kernels?1724527226931 -> /hub/user/[username]/api/kernels?1724527226931 (@127.0.0.1) 0.77ms Aug 24 19:20:27 myserver jupyterhub[790246]: [W 2024-08-24 19:20:27.244 JupyterHub base:1614] Failing suspected API request to not-running server: /hub/user/[username]/api/kernels Aug 24 19:20:27 myserver jupyterhub[790246]: [W 2024-08-24 19:20:27.244 JupyterHub log:192] 424 GET /hub/user/[username]/api/kernels?1724527226931 ([username]@127.0.0.1) 5.67ms Aug 24 19:20:27 myserver jupyterhub[790246]: [I 2024-08-24 19:20:27.421 JupyterHub log:192] 302 GET /user/[username]/api/sessions?1724527227371 -> /hub/user/[username]/api/sessions?1724527227371 (@127.0.0.1) 0.66ms Aug 24 19:20:27 myserver jupyterhub[790246]: [W 2024-08-24 19:20:27.516 JupyterHub base:1614] Failing suspected API request to not-running server: /hub/user/[username]/api/sessions ```
File `/etc/systemd/system/jupyterhub.service`: ``` [Unit] Description=JuputerHub [Service] Type=simple ExecStart=/[venv directory]/jupyterhub --port=8193 -f /[config directory]/jupyterhub_config.py User=root Group=root Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target ```
Configuration: ``` c = get_config() c.JupyterHub.spawner_class = "systemd" c.Spawner.notebook_dir = "/directory" c.SystemdSpawner.extra_paths = ["[venv directory]/bin"] c.SystemdSpawner.dynamic_users = True c.SystemdSpawner.readwrite_paths = ["/directory"] c.SystemdSpawner.disable_user_sudo = False c.SystemdSpawner.isolate_devices = True c.SystemdSpawner.user_workingdir = "/directory" c.JupyterHub.bind_url = 'http://127.0.0.1:8193' ```
Full environment ``` alembic==1.13.2 annotated-types==0.7.0 anyio==4.4.0 appdirs==1.4.4 argon2-cffi==23.1.0 argon2-cffi-bindings==21.2.0 arrow==1.3.0 asttokens==2.4.1 async-lru==2.0.4 attrs==24.2.0 babel==2.16.0 beautifulsoup4==4.12.3 bleach==6.1.0 certifi==2024.7.4 certipy==0.1.3 cffi==1.17.0 chardet==3.0.4 charset-normalizer==3.3.2 comm==0.2.2 contourpy==1.2.1 cryptography==43.0.0 cycler==0.12.1 DateTime==5.5 debugpy==1.8.5 decorator==5.1.1 defusedxml==0.7.1 deprecation==2.1.0 docker==7.1.0 dockerspawner==13.0.0 escapism==1.0.1 eventemitter==0.2.0 executing==2.0.1 fastjsonschema==2.20.0 fonttools==4.53.1 fqdn==1.5.1 greenlet==3.0.3 h11==0.14.0 h2==3.2.0 hpack==3.0.0 httpcore==1.0.5 httpx==0.27.0 hyperframe==5.2.0 idna==2.10 ipykernel==6.29.5 ipympl==0.9.4 ipython==8.26.0 ipython-genutils==0.2.0 ipywidgets==8.1.5 isoduration==20.11.0 jedi==0.19.1 Jinja2==3.1.4 json5==0.9.25 jsonpointer==3.0.0 jsonschema==4.23.0 jsonschema-specifications==2023.12.1 jupyter==1.0.0 jupyter-console==6.6.3 jupyter-events==0.10.0 jupyter-lsp==2.2.5 jupyter_client==8.6.2 jupyter_core==5.7.2 jupyter_server==2.14.2 jupyter_server_terminals==0.5.3 jupyterhub==5.1.0 jupyterhub-systemdspawner==1.0.1 jupyterlab==4.2.4 jupyterlab_pygments==0.3.0 jupyterlab_server==2.27.3 jupyterlab_widgets==3.0.13 kiwisolver==1.4.5 Mako==1.3.5 MarkupSafe==2.1.5 matplotlib==3.9.2 matplotlib-inline==0.1.7 mistune==3.0.2 nbclient==0.10.0 nbconvert==7.16.4 nbformat==5.10.4 nest-asyncio==1.6.0 notebook==7.2.1 notebook_shim==0.2.4 numpy==2.1.0 oauthlib==3.2.2 overrides==7.7.0 packaging==24.1 pamela==1.2.0 pandas==2.2.2 pandocfilters==1.5.1 parso==0.8.4 pexpect==4.9.0 pillow==10.4.0 platformdirs==4.2.2 prometheus_client==0.20.0 prompt_toolkit==3.0.47 psutil==6.0.0 ptyprocess==0.7.0 pure_eval==0.2.3 pycparser==2.22 pydantic==2.8.2 pydantic_core==2.23.0 Pygments==2.18.0 PyMySQL==1.1.1 pyOpenSSL==24.2.1 pyparsing==3.1.2 python-configuration==0.12.1 python-dateutil==2.9.0.post0 python-json-logger==2.0.7 pytz==2024.1 PyYAML==6.0.2 pyzmq==26.2.0 qtconsole==5.5.2 QtPy==2.4.1 referencing==0.35.1 requests==2.32.3 rfc3339-validator==0.1.4 rfc3986==1.5.0 rfc3986-validator==0.1.1 rpds-py==0.20.0 scipy==1.14.1 Send2Trash==1.8.3 setuptools==73.0.1 six==1.16.0 sniffio==1.3.1 soupsieve==2.6 SQLAlchemy==2.0.32 stack-data==0.6.3 terminado==0.18.1 tinycss2==1.3.0 tornado==6.4.1 traitlets==5.14.3 types-python-dateutil==2.9.0.20240821 typing_extensions==4.12.2 tzdata==2024.1 uri-template==1.3.0 urllib3==2.2.2 validators==0.33.0 watchdog==4.0.2 wcwidth==0.2.13 webcolors==24.8.0 webencodings==0.5.1 websocket-client==1.8.0 widgetsnbextension==4.0.13 zope.interface==7.0.1 ```
harvidsen commented 1 month ago

I have the same problem. i see

File "/nix/store/ng8wivb5xfpyygld64sf3a2w4xx2yaff-python3-3.12.5-env/lib/python3.12/site-packages/system>
         ensure_environment_directory(environment_file_directory)
       File "/nix/store/ng8wivb5xfpyygld64sf3a2w4xx2yaff-python3-3.12.5-env/lib/python3.12/site-packages/system>
         os.makedirs(environment_file_directory, mode=0o700, exist_ok=True)
       File "<frozen os>", line 225, in makedirs
     FileExistsError: [Errno 17] File exists: '/run/jupyter-harvidsen-singleuser'

when running on NixOS. Which seems weird since exist_ok=True. However, the files seems to be a symlink to somewhere i cannot find

sudo ls -la /run/jupyter-harvidsen-singleuser
lrwxrwxrwx 1 root root 36 Oct  9 12:15 /run/jupyter-harvidsen-singleuser -> private/jupyter-harvidsen-singleuser

So perhaps the issue lies in how the directory (/symlink) is created?