jupyterhub / the-littlest-jupyterhub

Simple JupyterHub distribution for 1-100 users on a single server
https://tljh.jupyter.org
BSD 3-Clause "New" or "Revised" License
1.03k stars 339 forks source link

Spawn failed: Server at http://127.0.0.1 #401

Open quantiota opened 5 years ago

quantiota commented 5 years ago

On a local server, cannot connect as admin or users. I am not sure but it seems this issue comes after installing configparser with sudo -E pip install configparser

error message:

Spawn failed: Server at http://127.0.0.1:42359/user/devbox/ didn't respond in 30 seconds
Event log
Server requested
Spawning server...
Spawn failed: Server at http://127.0.0.1:42359/user/devbox/ didn't respond in 30 seconds

the logs:

Jul 12 15:20:16 quantiota python3[1241]: [I 2019-07-12 15:20:16.148 JupyterHub log:174] 200 GET /hub/api/users/devbox/server/progress (devbox@192.168.0.254) 30931.27ms
Jul 12 15:21:52 quantiota python3[1241]: [I 2019-07-12 15:21:52.529 JupyterHub log:174] 302 GET /hub/ -> /hub/spawn (devbox@192.168.0.254) 20.00ms
Jul 12 15:21:52 quantiota python3[1241]: [I 2019-07-12 15:21:52.645 JupyterHub systemdspawner:217] user:devbox Unit jupyter-devbox in a failed state. Resetting state.
Jul 12 15:21:52 quantiota python3[1241]: Running as unit: jupyter-devbox.service
Jul 12 15:21:53 quantiota python3[1241]: [I 2019-07-12 15:21:53.567 JupyterHub log:174] 302 GET /hub/spawn -> /hub/spawn-pending/devbox (devbox@192.168.0.254) 1017.31ms
Jul 12 15:21:53 quantiota python3[1241]: [I 2019-07-12 15:21:53.608 JupyterHub pages:303] devbox is pending spawn
Jul 12 15:21:56 quantiota python3[1241]: [I 2019-07-12 15:21:56.024 JupyterHub proxy:319] Checking routes
Jul 12 15:22:02 quantiota python3[1241]: ERROR:asyncio:Task exception was never retrieved
Jul 12 15:22:02 quantiota python3[1241]: future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py:697> exception=HTTPError()>
Jul 12 15:22:02 quantiota python3[1241]: Traceback (most recent call last):
Jul 12 15:22:02 quantiota python3[1241]:   File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user
Jul 12 15:22:02 quantiota python3[1241]:     timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
Jul 12 15:22:02 quantiota python3[1241]: tornado.util.TimeoutError: Timeout
Jul 12 15:22:02 quantiota python3[1241]: During handling of the above exception, another exception occurred:
Jul 12 15:22:02 quantiota python3[1241]: Traceback (most recent call last):
Jul 12 15:22:02 quantiota python3[1241]:   File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 922, in spawn_single_user
Jul 12 15:22:02 quantiota python3[1241]:     % (status, spawner._log_name),
Jul 12 15:22:02 quantiota python3[1241]: tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for devbox may contain details.)
Jul 12 15:22:21 quantiota python3[1241]: [W 2019-07-12 15:22:21.321 JupyterHub user:678] devbox's server never showed up at http://127.0.0.1:42359/user/devbox/ after 30 seconds. Giving up
Jul 12 15:22:21 quantiota python3[1241]: [E 2019-07-12 15:22:21.372 JupyterHub gen:593] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py:800> exception=TimeoutError("Server at http://127.0.0.1:42359/user/devbox/ didn't respond in 30 seconds",)> after timeout
Jul 12 15:22:21 quantiota python3[1241]:     Traceback (most recent call last):
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/tornado/gen.py", line 589, in error_callback
Jul 12 15:22:21 quantiota python3[1241]:         future.result()
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
Jul 12 15:22:21 quantiota python3[1241]:         await spawn_future
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/user.py", line 654, in spawn
Jul 12 15:22:21 quantiota python3[1241]:         await self._wait_up(spawner)
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/user.py", line 701, in _wait_up
Jul 12 15:22:21 quantiota python3[1241]:         raise e
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/user.py", line 669, in _wait_up
Jul 12 15:22:21 quantiota python3[1241]:         http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
Jul 12 15:22:21 quantiota python3[1241]:         timeout=timeout,
Jul 12 15:22:21 quantiota python3[1241]:       File "/opt/tljh/hub/lib/python3.6/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
Jul 12 15:22:21 quantiota python3[1241]:         raise TimeoutError(fail_message)
Jul 12 15:22:21 quantiota python3[1241]:     TimeoutError: Server at http://127.0.0.1:42359/user/devbox/ didn't respond in 30 seconds
Jul 12 15:22:21 quantiota python3[1241]:
Jul 12 15:22:21 quantiota python3[1241]: [I 2019-07-12 15:22:21.374 JupyterHub log:174] 200 GET /hub/api/users/devbox/server/progress (devbox@192.168.0.254) 27538.02ms
lachlancampbell commented 5 years ago

This is the same issue I'm having #351 Is there a way to further diagnose this? The logs aren't showing me anything, but can we get a verbose output from the single-user notebook server, as this appears to be where the problem is.

ghost commented 5 years ago

Have no idea what you are talking about.. but you might know how to connect to the router set on 127.0.0.1/8 network .. please let me know

quantiota commented 5 years ago

i restored the tljh from a daily backup and now it works.

Adrianhein commented 4 years ago

This is the same issue I'm having #351 Is there a way to further diagnose this? The logs aren't showing me anything, but can we get a verbose output from the single-user notebook server, as this appears to be where the problem is.

Hello I also same issue in trouble ,let me know how to solve this .

Adrianhein commented 4 years ago

Hi All,

I have run the jupyterhub in Single user .But when I open in browser timeout error show .Please help and check the logs.

[W 2019-12-23 17:18:35.594 JupyterHub auth:941] Failed to open PAM session for user-name: [PAM Error 14] Cannot make/remove an entry for the specified session
[W 2019-12-23 17:18:35.594 JupyterHub auth:942] Disabling PAM sessions from now on.
[I 2019-12-23 17:18:35.601 JupyterHub spawner:1387] Spawning jupyterhub-singleuser --port=34684
Traceback (most recent call last):
File "/usr/local/bin/jupyterhub-singleuser", line 7, in
from jupyterhub.singleuser import main
File "/usr/local/lib/python3.6/site-packages/jupyterhub/singleuser.py", line 14, in
from jinja2 import ChoiceLoader
ModuleNotFoundError: No module named 'jinja2'
[I 2019-12-23 17:18:36.567 JupyterHub log:174] 302 GET /hub/spawn/user-name -> /hub/spawn-pending/user-name 1011.63ms
[I 2019-12-23 17:18:36.578 JupyterHub pages:303] user-name is pending spawn
[I 2019-12-23 17:18:36.581 JupyterHub log:174] 200 GET /hub/spawn-pending/user-name 10.55ms
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py:697> exception=HTTPError()>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user
timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future
tornado.util.TimeoutError: Timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 922, in spawn_single_user
% (status, spawner._log_name),
tornado.web.HTTPError: HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for user-name may contain details.)
[W 2019-12-23 17:19:06.798 JupyterHub user:678] user-name's server never showed up at http://127.0.0.1:34684/user/user-name/ after 30 seconds. Giving up
[E 2019-12-23 17:19:06.816 JupyterHub gen:599] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user..finish_user_spawn() done, defined at /usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py:800> exception=TimeoutError("Server at http://127.0.0.1:34684/user/user-name/ didn't respond in 30 seconds",)> after timeout
Traceback (most recent call last):
File "/usr/local/lib64/python3.6/site-packages/tornado/gen.py", line 593, in error_callback
future.result()
File "/usr/local/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn
await spawn_future
File "/usr/local/lib/python3.6/site-packages/jupyterhub/user.py", line 654, in spawn
await self._wait_up(spawner)
File "/usr/local/lib/python3.6/site-packages/jupyterhub/user.py", line 701, in _wait_up
raise e
File "/usr/local/lib/python3.6/site-packages/jupyterhub/user.py", line 669, in _wait_up
http=True, timeout=spawner.http_timeout, ssl_context=ssl_context
File "/usr/local/lib/python3.6/site-packages/jupyterhub/utils.py", line 234, in wait_for_http_server
timeout=timeout,
File "/usr/local/lib/python3.6/site-packages/jupyterhub/utils.py", line 177, in exponential_backoff
raise TimeoutError(fail_message)
TimeoutError: Server at http://127.0.0.1:34684/user/user-name/ didn't respond in 30 seconds

[I 2019-12-23 17:19:06.817 JupyterHub log:174] 200 GET /hub/api/users/user-name/server/progress 30191.89ms
mukhendra commented 4 years ago

Hello All,

I'm also facing the similar issue can you please suggest how to resolve this. " your server is starting up and you will be redirected automaticaly when it's ready for you." Spawn got failed and it's pending.

subgero commented 4 years ago

@mukhendra Im facing the same problem with jupyterhub for multiuser in a docker env, let me know if you found a solution for this, I've googled but without success til now.

jesusmariajurado commented 4 years ago

Hi all, for me this (https://github.com/jupyterhub/jupyterhub/issues/2600#issuecomment-587985814) works

stephen-a2z commented 4 years ago

facing the same problem

yashag66 commented 4 years ago

@subgero @GeorgianaElena @stephen-a2z I'm still facing the same issue, how did you resolve this issue guys? I tried following this link stated by @jesusmariajurado (https://github.com/jupyterhub/the-littlest-jupyterhub/issues/401#issuecomment-610774960) and it didn't helped.

subgero commented 4 years ago

@yashag66 I cant remember but this is how it works for me

Dockerfile

FROM jupyterhub/jupyterhub
RUN apt update -y
RUN apt upgrade -y
RUN apt clean -y
RUN apt install mlocate

RUN umask 002

RUN apt remove python3 -y
RUN apt install python3.7 -y

RUN update-alternatives --install /usr/local/bin/python3 python3 /usr/bin/python3.6 1
RUN update-alternatives --install /usr/local/bin/python3 python3 /usr/bin/python3.7 2
RUN update-alternatives --config python3
RUN update-alternatives  --set python3 /usr/bin/python3.7

RUN apt install python3-pip -y

RUN python3.7 -m pip install pip
RUN python3.7 -m pip install ipykernel
RUN python3.7 -m pip install --upgrade pip setuptools
RUN python3.7 -m pip install --upgrade requests
RUN python3.7 -m pip install sudospawner
RUN python3.7 -m pip install oauthenticator
RUN python3.7 -m pip install ipyparallel
RUN python3.7 -m pip install jupyter

RUN ipython kernelspec install-self
COPY src/conf/requirements.txt /srv/jupyterhub/requirements.txt
COPY kernel.json /usr/local/share/jupyter/kernels/python3/kernel.json

requirements.txt

-i https://pypi.org/simple
attrs==19.3.0
backcall==0.1.0
cachetools==4.1.0
certifi==2020.4.5.1
chardet==3.0.4
click-plugins==1.1.1
click==7.1.2
cligj==0.5.0
couchdb==1.2
cycler==0.10.0
decorator==4.4.2
deprecation==2.1.0
descartes==1.1.0
elasticsearch-dsl==7.2.0
elasticsearch==7.7.1
fiona==1.8.13.post1
geopandas==0.7.0
google-api-core==1.17.0
google-api-python-client==1.8.4
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-auth==1.16.0
googleapis-common-protos==1.51.0
httplib2==0.18.1
idna==2.9
ipykernel==5.3.0
ipython-genutils==0.2.0
ipython==7.14.0
jedi==0.17.0
jupyter-client==6.1.3
jupyter-core==4.6.3
kiwisolver==1.2.0
matplotlib==3.2.1
munch==2.5.0
networkx==2.4
numpy==1.18.4
oauthlib==3.1.0
packaging==20.4
pandas==1.0.4
parso==0.7.0
pexpect==4.8.0 ; sys_platform != 'win32'
pickleshare==0.7.5
progressbar2==3.51.3
prompt-toolkit==3.0.5
protobuf==3.12.2
ptyprocess==0.6.0
pyasn1-modules==0.2.8
pyasn1==0.4.8
pygments==2.6.1
pyparsing==2.4.7
pyproj==2.6.1.post1
python-dateutil==2.8.1
python-utils==2.4.0
pytz==2020.1
pyzmq==19.0.1
requests-oauthlib==1.3.0
requests==2.23.0
rsa==4.0
shapely==1.7.0
six==1.15.0
sqlalchemy==1.3.17
tornado==6.0.4
traitlets==4.3.3
uritemplate==3.0.1
urllib3==1.25.9
wcwidth==0.1.9
xlrd==1.2.0
xlsxwriter==1.2.8

kernel.json

{
 "argv": [
  "/usr/local/bin/python3",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "Python 3",
 "language": "python"
}

docker-compose.yml

version: '2'
services:
  jupyterhubkingo:
    ports:
      - "8000:8000"
    image: "jupyterhubkingo"
    container_name: jupyterhubkingo
    restart: always

Then, inside of container you run this

pip3 install -r requirements.txt

I hope it helps you.

mjmanas0699 commented 3 years ago

1.-> docker exec -it jupyterhub bash
2.-> pip install notebook hope it worked

zatolokin commented 3 years ago

Hello, I faced the same situation with JupyterHub. The solution was to get moe disk space.

zqszzs commented 3 years ago

hi all, I'm facing the same issue, the notebook server container has started, but jupyterhub also say start timeout. how did you solve this? image

subgero commented 3 years ago

Hi @zqszzs , I cant remember how solved this and Im not able to login in the project anymore, but you can try to create a network between your service in your docker-file, or instead of use 127.0.0.1 you can use the Internal IP of your instance like: 10.0.2.1

amirhosseinkabiri commented 7 months ago

pooof! I just solved this crazy problem. I used to login with root user and even installed the anaconda using root user in /root directory. but when I used another VM, created another user name condauser, installed the anaconda with that user, every thing works fine. remind that you should run the jupyterhub with root user but use that recently created user to login.