search

jupyterhub / systemdspawner

Spawn JupyterHub single-user notebook servers with systemd
BSD 3-Clause "New" or "Revised" License
92 stars 45 forks source link

Spawner failed to start #57

Closed w0lfvn closed 3 years ago

w0lfvn commented 5 years ago

I installed JupyterHub on CentOS 7 and config the spawner using sudospawner. I can access to the web interface but the spawner failed to start. Does anyone know what's wrong here?

[D 2019-07-10 19:57:08.470 JupyterHub base:289] Refreshing auth for w0lfvn
[D 2019-07-10 19:57:08.471 JupyterHub user:240] Creating <class 'systemdspawner.systemdspawner.SystemdSpawner'> for w0lfvn:
[D 2019-07-10 19:57:08.478 JupyterHub systemdspawner:96] user:w0lfvn Initialized spawner with unit jupyter-w0lfvn-singleuser
[I 2019-07-10 19:57:08.479 JupyterHub log:174] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn (w0lfvn@14.161.31.200) 33.27ms
[D 2019-07-10 19:57:08.514 JupyterHub pages:165] Triggering spawn with default options for w0lfvn
[D 2019-07-10 19:57:08.514 JupyterHub base:780] Initiating spawn for w0lfvn
[D 2019-07-10 19:57:08.515 JupyterHub base:787] 0/100 concurrent spawns
[D 2019-07-10 19:57:08.515 JupyterHub base:792] 0 active servers
[W 2019-07-10 19:57:08.956 JupyterHub auth:941] Failed to open PAM session for w0lfvn: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2019-07-10 19:57:08.956 JupyterHub auth:942] Disabling PAM sessions from now on.
[D 2019-07-10 19:57:08.977 JupyterHub user:542] Calling Spawner.start for w0lfvn
[D 2019-07-10 19:57:08.979 JupyterHub systemdspawner:144] Bad message (ValueError("unsupported format character 'U' (0x55) at index 7",)): {'name': 'JupyterHub', 'msg': 'user:% Using port %s to start spawning for user %s', 'args': ('w0lfvn', 36034), 'levelname': 'DEBUG', 'levelno': 10, 'pathname': '/usr/lib/python3.6/site-packages/systemdspawner/systemdspawner.py', 'filename': 'systemdspawner.py', 'module': 'systemdspawner', 'exc_info': None, 'exc_text': None, 'stack_info': None, 'lineno': 144, 'funcName': 'start', 'created': 1562763428.9799898, 'msecs': 979.989767074585, 'relativeCreated': 8066.693067550659, 'thread': 140526989694784, 'threadName': 'MainThread', 'processName': 'MainProcess', 'process': 6626}
[I 2019-07-10 19:57:09.036 JupyterHub systemdspawner:160] user:w0lfvn Unit jupyter-w0lfvn-singleuser in failed state, resetting
[D 2019-07-10 19:57:09.039 JupyterHub systemdspawner:240] user:w0lfvn Running systemd-run with: /usr/bin/systemd-run --unit jupyter-w0lfvn-singleuser --uid 1104 --gid 1104 --setenv=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin --setenv=LANG=en_US.UTF-8 --setenv=JUPYTERHUB_API_TOKEN=cf1284755a9a4d858976cece7887b70e --setenv=JPY_API_TOKEN=cf1284755a9a4d858976cece7887b70e --setenv=JUPYTERHUB_CLIENT_ID=jupyterhub-user-w0lfvn --setenv=JUPYTERHUB_HOST= --setenv=JUPYTERHUB_OAUTH_CALLBACK_URL=/jupyter/user/w0lfvn/oauth_callback --setenv=JUPYTERHUB_USER=w0lfvn --setenv=JUPYTERHUB_SERVER_NAME= --setenv=JUPYTERHUB_API_URL=http://phoenix.data:9091/jupyter/hub/api --setenv=JUPYTERHUB_ACTIVITY_URL=http://phoenix.data:9091/jupyter/hub/api/users/w0lfvn/activity --setenv=JUPYTERHUB_BASE_URL=/jupyter/ --setenv=JUPYTERHUB_SERVICE_PREFIX=/jupyter/user/w0lfvn/ --setenv=SHELL=/bin/bash /bin/bash -c cd /home/w0lfvn && exec jupyterhub-singleuser --port=36034 '--notebook-dir=~/notebooks' --NotebookApp.default_url=/tree
Running as unit jupyter-w0lfvn-singleuser.service.
[D 2019-07-10 19:57:09.068 JupyterHub systemdspawner:265] user:w0lfvn unit jupyter-w0lfvn-singleuser is active
[D 2019-07-10 19:57:09.084 JupyterHub spawner:1084] Polling subprocess every 30s
[I 2019-07-10 19:57:09.517 JupyterHub log:174] 302 GET /jupyter/hub/spawn -> /jupyter/hub/spawn-pending/w0lfvn (w0lfvn@14.161.31.200) 1019.93ms
[I 2019-07-10 19:57:09.562 JupyterHub pages:303] w0lfvn is pending spawn
[I 2019-07-10 19:57:09.629 JupyterHub log:174] 200 GET /jupyter/hub/spawn-pending/w0lfvn (w0lfvn@14.161.31.200) 87.84ms
[D 2019-07-10 19:57:18.535 JupyterHub systemdspawner:268] user:w0lfvn unit jupyter-w0lfvn-singleuser is not active
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /usr/lib/python3.6/site-packages/jupyterhub/handlers/base.py:697> exception=HTTPError()>
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user
    timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
tornado.util.TimeoutError: Timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 922, in spawn_single_user
    % (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=3]. The logs for w0lfvn may contain details.)
[W 2019-07-10 19:57:36.370 JupyterHub user:678] w0lfvn's server never showed up at http://127.0.0.1:36034/jupyter/user/w0lfvn/ after 30 seconds. Giving up
[D 2019-07-10 19:57:36.371 JupyterHub user:724] Stopping w0lfvn
[D 2019-07-10 19:57:36.387 JupyterHub systemdspawner:268] user:w0lfvn unit jupyter-w0lfvn-singleuser is not active
[D 2019-07-10 19:57:36.407 JupyterHub user:752] Deleting oauth client jupyterhub-user-w0lfvn
[D 2019-07-10 19:57:36.430 JupyterHub user:755] Finished stopping w0lfvn
[E 2019-07-10 19:57:36.445 JupyterHub gen:974] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/lib/python3.6/site-packages/jupyterhub/handlers/base.py:800> exception=TimeoutError("Server at http://127.0.0.1:36034/jupyter/user/w0lfvn/ didn't respond in 30 seconds",)> after timeout
    Traceback (most recent call last):
      File "/usr/lib64/python3.6/site-packages/tornado/gen.py", line 970, in error_callback
        future.result()
      File "/usr/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
        await spawn_future
      File "/usr/lib/python3.6/site-packages/jupyterhub/user.py", line 654, in spawn
        await self._wait_up(spawner)
      File "/usr/lib/python3.6/site-packages/jupyterhub/user.py", line 701, in _wait_up
        raise e
      File "/usr/lib/python3.6/site-packages/jupyterhub/user.py", line 669, in _wait_up
        http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
      File "/usr/lib/python3.6/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
        timeout=timeout,
      File "/usr/lib/python3.6/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://127.0.0.1:36034/jupyter/user/w0lfvn/ didn't respond in 30 seconds
Yensan commented 5 years ago

Same issue.

jakepaw commented 5 years ago

Same with me. Any updates on this from anyone?

TuSKan commented 5 years ago

Same here. Any help?

charnpreetsingh commented 5 years ago

https://github.com/jupyterhub/systemdspawner/pull/53 Has the changes that fix this issue

jakepaw commented 5 years ago

I agree with @charnpreetsingh that this is the correct fix. I also would ensure that your conda / pip packages are able to work together such as the correct Tornado package.

consideRatio commented 3 years ago

Identified to be closed by #53 for @charnpreetsingh - thanks!