My jupyterhub database had a PID entry for a user notebook server that no longer existed. When the hub started up and tried to health check that PID, it hit the following exception and failed to start. The only workaround was to nuke the database or perform surgery on it to remove the offending state.
Traceback (most recent call last):
File "/opt/jupyterhub/envs/jupyterhub/bin/sudospawner", line 4, in <module>
mediator.main()
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/mediator.py", line 148, in main
kill(**kwargs)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/mediator.py", line 52, in kill
os.kill(pid, signal)
PermissionError: [Errno 1] Operation not permitted
ESC[31m[E 2017-04-17 20:40:54.603 JupyterHub app:1527]ESC[0;10m
Traceback (most recent call last):
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/app.py", line 1524, in launch_instance_async
yield self.initialize(argv)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/types.py", line 179, in throw
return self.__wrapped.throw(tp, *rest)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/app.py", line 1315, in initialize
yield self.init_spawners()
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/app.py", line 1087, in init_spawners
status = yield spawner.poll()
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/jupyterhub/spawner.py", line 834, in poll
alive = yield self._signal(0)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/spawner.py", line 90, in _signal
reply = yield self.do('kill', pid=self.pid, signal=sig)
File "/opt/jupyterhub/envs/jupyterhub/lib/python3.5/site-packages/sudospawner/spawner.py", line 69, in do
data_str = data_str[data_str.index('{'):data_str.rindex('}')+1]
ValueError: substring not found
I'm not sure how it got into the state that it did. Maybe the notebook server went away while the hub was restarting? Regardless, the hub should be robust against the situation and at least start.
Should be fixed by #33. Something is definitely wrong to get into this state, but it shouldn't cause the whole thing to give up. The poor error message should also be fixed by #31.
My jupyterhub database had a PID entry for a user notebook server that no longer existed. When the hub started up and tried to health check that PID, it hit the following exception and failed to start. The only workaround was to nuke the database or perform surgery on it to remove the offending state.
I'm not sure how it got into the state that it did. Maybe the notebook server went away while the hub was restarting? Regardless, the hub should be robust against the situation and at least start.