search

jupyterhub / batchspawner

Custom Spawner for Jupyterhub to start servers in batch scheduled systems
BSD 3-Clause "New" or "Revised" License
190 stars 134 forks source link

404 Trying to connect to single-user notebook page #222

Open ivhacks opened 3 years ago

ivhacks commented 3 years ago

(Note: I've removed my username, domain name, and indentifiable IP addresses just to be safe.)

Bug description

After the SLURM process is started, the user is directed to a 404 page.

Expected behaviour

Actual behaviour

How to reproduce

This happens every time I try to use Jupyterhub. Probably system-specific, because it's unavoidable.

Setup

Other relevant info:

Logs

Full environment

``` XDG_SESSION_ID=182 HOSTNAME=jupyterhub-test SELINUX_ROLE_REQUESTED= __LMOD_REF_COUNT_MODULEPATH=/opt/ohpc/pub/modulefiles:1 TERM=xterm-256color SHELL=/bin/bash HISTSIZE=1000 SSH_CLIENT=aaa.aaa.aaa.aaa 63602 22 CONDA_SHLVL=1 CONDA_PROMPT_MODIFIER=(jupyterenv) SELINUX_USE_CURRENT_RANGE= LMOD_PKG=/opt/ohpc/admin/lmod/lmod OLDPWD=/home/iv LMOD_VERSION=7.8.1 SSH_TTY=/dev/pts/0 USER=iv LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45: CONDA_EXE=/usr/local/anaconda3/bin/conda _CE_CONDA= LMOD_PREPEND_BLOCK=normal _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT1mYWxzZSxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXt9LG1wYXRoQT17Ii9vcHQvb2hwYy9wdWIvbW9kdWxlZmlsZXMiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL29wdC9vaHBjL3B1Yi9tb2R1bGVmaWxlcyIsfQ== MAIL=/var/spool/mail/iv PATH=/home/iv/.conda/envs/jupyterenv/bin:/usr/local/anaconda3/condabin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/iv/.local/bin:/home/iv/bin CONDA_PREFIX=/home/iv/.conda/envs/jupyterenv LMOD_SETTARG_CMD=: PWD=/home/iv/jupyterhub LANG=en_US.UTF-8 MODULEPATH=/opt/ohpc/pub/modulefiles _ModuleTable_Sz_=1 SELINUX_LEVEL_REQUESTED= LMOD_CMD=/opt/ohpc/admin/lmod/lmod/libexec/lmod _CE_M= HISTCONTROL=ignoredups SHLVL=1 HOME=/home/iv BASH_ENV=/opt/ohpc/admin/lmod/lmod/init/bash CONDA_PYTHON_EXE=/usr/local/anaconda3/bin/python LOGNAME=iv SSH_CONNECTION=aaa.aaa.aaa.aaa 63602 aaa.aaa.aaa.aaa 22 MODULESHOME=/opt/ohpc/admin/lmod/lmod CONDA_DEFAULT_ENV=jupyterenv LMOD_SETTARG_FULL_SUPPORT=no LESSOPEN=||/usr/bin/lesspipe.sh %s LMOD_FULL_SETTARG_SUPPORT=no XDG_RUNTIME_DIR=/run/user/1002 LMOD_DIR=/opt/ohpc/admin/lmod/lmod/libexec LMOD_COLORIZE=no BASH_FUNC_module()=() { eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh) } BASH_FUNC_ml()=() { eval $($LMOD_DIR/ml_cmd "$@") } _=/usr/bin/env ```

conda list output

``` (jupyterenv) [iv@jupyterhub-test jupyterhub]$ conda list # packages in environment at /home/iv/.conda/envs/jupyterenv: # # Name Version Build Channel _libgcc_mutex 0.1 main alembic 1.6.4 pyhd8ed1ab_0 conda-forge anyio 2.2.0 py39h06a4308_1 argon2-cffi 20.1.0 py39h27cfd23_1 async_generator 1.10 pyhd3eb1b0_0 attrs 21.2.0 pyhd3eb1b0_0 babel 2.9.1 pyhd3eb1b0_0 backcall 0.2.0 pyhd3eb1b0_0 batchspawner 1.1.0 pypi_0 pypi bleach 3.3.0 pyhd3eb1b0_0 blinker 1.4 py_1 conda-forge brotlipy 0.7.0 py39h27cfd23_1003 ca-certificates 2020.12.5 ha878542_0 conda-forge certifi 2020.12.5 py39hf3d152e_1 conda-forge certipy 0.1.3 py_0 conda-forge cffi 1.14.5 py39h261ae71_0 chardet 4.0.0 py39h06a4308_1003 configurable-http-proxy 4.3.2 node12_hac6e442_0 conda-forge cryptography 3.4.7 py39hd23ed53_0 decorator 5.0.9 pyhd3eb1b0_0 defusedxml 0.7.1 pyhd3eb1b0_0 elementpath 2.2.2 pypi_0 pypi entrypoints 0.3 py39h06a4308_0 icu 68.1 h58526e2_0 conda-forge idna 2.10 pyhd3eb1b0_0 importlib-metadata 3.10.0 py39h06a4308_0 importlib-resources 5.1.2 pypi_0 pypi importlib_metadata 3.10.0 hd3eb1b0_0 iniconfig 1.1.1 pypi_0 pypi ipykernel 5.3.4 py39hb070fc8_0 ipython 7.22.0 py39hb070fc8_0 ipython_genutils 0.2.0 pyhd3eb1b0_1 jedi 0.17.2 py39h06a4308_1 jinja2 3.0.0 pyhd3eb1b0_0 json5 0.9.5 py_0 jsonschema 3.2.0 py_2 jupyter-packaging 0.7.12 pyhd3eb1b0_0 jupyter-saml2authenticator 0.1.0.dev0 pypi_0 pypi jupyter_client 6.1.12 pyhd3eb1b0_0 jupyter_core 4.7.1 py39h06a4308_0 jupyter_server 1.4.1 py39h06a4308_0 jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge jupyterhub 1.4.1 py39hf3d152e_0 conda-forge jupyterhub-base 1.4.1 py39hf3d152e_0 conda-forge jupyterlab 3.0.14 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.1.2 py_0 jupyterlab_server 2.4.0 pyhd3eb1b0_0 krb5 1.18.2 h173b8e3_0 ld_impl_linux-64 2.33.1 h53a641e_7 libcurl 7.71.1 h20c2e04_1 libedit 3.1.20191231 he28a2e2_2 conda-forge libffi 3.3 he6710b0_2 libgcc-ng 9.1.0 hdf63c60_0 libsodium 1.0.18 h7b6447c_0 libssh2 1.9.0 hab1572f_5 conda-forge libstdcxx-ng 9.1.0 hdf63c60_0 libuv 1.40.0 h7b6447c_0 mako 1.1.4 pyh44b312d_0 conda-forge markupsafe 2.0.1 py39h27cfd23_0 mistune 0.8.4 py39h27cfd23_1000 nbclassic 0.2.6 pyhd3eb1b0_0 nbclient 0.5.3 pyhd3eb1b0_0 nbconvert 6.0.7 py39h06a4308_0 nbformat 5.1.3 pyhd3eb1b0_0 ncurses 6.2 he6710b0_1 nest-asyncio 1.5.1 pyhd3eb1b0_0 nodejs 12.19.0 hfa01f41_2 conda-forge notebook 6.4.0 py39h06a4308_0 oauthlib 3.0.1 py_0 conda-forge openssl 1.1.1k h27cfd23_0 packaging 20.9 pyhd3eb1b0_0 pamela 1.0.0 py_0 conda-forge pandoc 2.12 h06a4308_0 pandocfilters 1.4.3 py39h06a4308_1 parso 0.7.0 py_0 pexpect 4.8.0 pyhd3eb1b0_3 pickleshare 0.7.5 pyhd3eb1b0_1003 pip 21.1.1 py39h06a4308_0 pluggy 0.13.1 pypi_0 pypi prometheus_client 0.10.1 pyhd3eb1b0_0 prompt-toolkit 3.0.17 pyh06a4308_0 ptyprocess 0.7.0 pyhd3eb1b0_2 py 1.10.0 pypi_0 pypi pycparser 2.20 py_2 pycurl 7.43.0.6 py39h8cd828d_0 conda-forge pygments 2.9.0 pyhd3eb1b0_0 pyjwt 2.1.0 pyhd8ed1ab_0 conda-forge pyopenssl 20.0.1 pyhd3eb1b0_1 pyparsing 2.4.7 pyhd3eb1b0_0 pyrsistent 0.17.3 py39h27cfd23_0 pysaml2 6.5.1 pypi_0 pypi pysocks 1.7.1 py39h06a4308_0 pytest 6.2.4 pypi_0 pypi python 3.9.4 hdb3f193_0 python-dateutil 2.8.1 pyhd3eb1b0_0 python-editor 1.0.4 py_0 conda-forge python-json-logger 2.0.1 pyh9f0ad1d_0 conda-forge python_abi 3.9 1_cp39 conda-forge pytz 2021.1 pyhd3eb1b0_0 pyzmq 20.0.0 py39h2531618_1 readline 8.1 h27cfd23_0 requests 2.25.1 pyhd3eb1b0_0 ruamel.yaml 0.16.12 py39h27cfd23_1 ruamel.yaml.clib 0.2.2 py39hbd71b63_1 conda-forge send2trash 1.5.0 pyhd3eb1b0_1 setuptools 52.0.0 py39h06a4308_0 six 1.15.0 py39h06a4308_0 sniffio 1.2.0 py39h06a4308_1 sqlalchemy 1.3.23 py39h27cfd23_0 sqlite 3.35.4 hdfb4753_0 terminado 0.9.4 py39h06a4308_0 testpath 0.4.4 pyhd3eb1b0_0 tk 8.6.10 hbc83047_0 toml 0.10.2 pypi_0 pypi tornado 6.1 py39h27cfd23_0 traitlets 5.0.5 pyhd3eb1b0_0 tzdata 2020f h52ac0ba_0 urllib3 1.26.4 pyhd3eb1b0_0 wcwidth 0.2.5 py_0 webencodings 0.5.1 py39h06a4308_1 wheel 0.36.2 pyhd3eb1b0_0 wrapspawner 1.0.0 pypi_0 pypi xmlschema 1.6.2 pypi_0 pypi xz 5.2.5 h7b6447c_0 zeromq 4.3.4 h2531618_0 zipp 3.4.1 pyhd3eb1b0_0 zlib 1.2.11 h7b6447c_3 ```

Configuration ```python c.JupyterHub.bind_url = 'https://[jupyterhub-test.contoso.com]:8000/' c.JupyterHub.cookie_secret_file = 'jupyterhub_cookie_secret' c.JupyterHub.hub_ip = 'jupyterhub-test.contoso.com' c.JupyterHub.ip = '' c.JupyterHub.port = 8000 import batchspawner.api c = get_config() c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner' c.Spawner.http_timeout = 300 c.Spawner.start_timeout = 300 c.BatchSpawnerBase.req_host = 'jupyterhub-test.contoso.com' c.BatchSpawnerBase.req_nprocs = '2' c.BatchSpawnerBase.req_memory = '2gb' c.BatchSpawnerBase.req_runtime = '6:00:00' c.TorqueSpawner.state_exechost_exp = r'in-\1.jupyterhub-test.contoso.com' c.ProfilesSpawner.profiles = [ ('2 cores/2 GB/6 hours', '2c2gb', 'batchspawner.SlurmSpawner', dict(req_nprocs='2', req_queue='normal', req_runtime='6:00:00', req_memory='2gb')), ('2 cores/8 GB/2 hours', '2c8gb', 'batchspawner.SlurmSpawner', dict(req_nprocs='2', req_queue='normal', req_runtime='2:00:00', req_memory='8gb')) ] c.JupyterHub.ssl_cert = '/opt/jupyterhub/jupyterhub-test.contoso.com.cer' c.JupyterHub.ssl_key = '/opt/jupyterhub/jupyterhub-test.contoso.com.key' from jupyter_saml2authenticator import Saml2Authenticator c.JupyterHub.authenticator_class = Saml2Authenticator c.Saml2Authenticator.saml2_metadata_url = *Prefer not to share* c.Saml2Authenticator.saml2_entity_id = 'https://jupyterhub-test.contoso.com/saml2_auth/ent' c.Saml2Authenticator.saml2_attribute_username = 'emailAddress' c.Saml2Authenticator.login_service = "Contoso Single Sign-on" c.Saml2Authenticator.saml2_strip_username = False c.Saml2Authenticator.saml2_login_url = r'/saml2_auth/login' c.Saml2Authenticator.saml2_acs_url = r'/saml2_auth/acs' ```
Logs This is the log from the compute node that the single-user notebook app runs on (errors at the end are from pressing ctrl+c): ``` /usr/local/anaconda3/bin/jupyterhub-singleuser [I 2021-06-12 11:24:16.326 SingleUserNotebookApp notebookapp:1593] Authentication of /metrics is OFF, since other authentication is disabled. [I 2021-06-12 11:24:17.826 SingleUserNotebookApp handlers:78] Loading IPython parallel extension [I 2021-06-12 11:24:20.378 LabApp] JupyterLab extension loaded from /usr/local/anaconda3/lib/python3.9/site-packages/jupyterlab [I 2021-06-12 11:24:20.379 LabApp] JupyterLab application directory is /usr/local/anaconda3/share/jupyter/lab Patching auth into jupyter_server.base.handlers.JupyterHandler(jupyter_server.base.handlers.AuthenticatedHandler) -> JupyterHandler(jupyterhub.singleuser.mixins.HubAuthenticatedHandler, jupyter_server.base.handlers.AuthenticatedHandler) [I 2021-06-12 11:24:20.392 SingleUserNotebookApp mixins:576] Starting jupyterhub-singleuser server version 1.4.1 [I 2021-06-12 11:24:20.399 SingleUserNotebookApp log:189] 302 GET /user/aerobinsoniv@contoso.com/ -> /user/aerobinsoniv@contoso.com (@aaa.aaa.aaa.aaa) 3.08ms [I 2021-06-12 11:24:20.409 SingleUserNotebookApp notebookapp:2302] Serving notebooks from local directory: /home/arobinson3@admin.contoso.com [I 2021-06-12 11:24:20.410 SingleUserNotebookApp notebookapp:2302] Jupyter Notebook 6.3.0 is running at: [I 2021-06-12 11:24:20.410 SingleUserNotebookApp notebookapp:2302] http://jupyterhub-c5:34582/ [I 2021-06-12 11:24:20.410 SingleUserNotebookApp notebookapp:2303] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [I 2021-06-12 11:24:20.442 SingleUserNotebookApp mixins:556] Updating Hub with activity every 300 seconds [I 2021-06-12 11:24:20.767 SingleUserNotebookApp log:189] 302 GET /user/aerobinsoniv@contoso.com/ -> /user/aerobinsoniv@contoso.com (@aaa.aaa.aaa.aaa) 1.52ms [W 2021-06-12 11:24:20.992 SingleUserNotebookApp log:189] 404 GET /user/aerobinsoniv@contoso.com (@aaa.aaa.aaa.aaa) 102.75ms [I 2021-06-12 12:14:13.269 SingleUserNotebookApp log:189] 302 GET /user/aerobinsoniv@contoso.com/ -> /user/aerobinsoniv@contoso.com (@aaa.aaa.aaa.aaa) 2.12ms [W 2021-06-12 12:14:13.458 SingleUserNotebookApp log:189] 404 GET /user/aerobinsoniv@contoso.com (@aaa.aaa.aaa.aaa) 8.65ms [W 2021-06-12 12:14:38.003 SingleUserNotebookApp log:189] 404 GET /user/aerobinsoniv@contoso.com (@aaa.aaa.aaa.aaa) 4.26ms srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: *** JOB 187 ON jupyterhub-c5 CANCELLED AT 2021-06-12T14:37:50 *** slurmstepd: error: *** STEP 187.0 ON jupyterhub-c5 CANCELLED AT 2021-06-12T14:37:50 *** [C 2021-06-12 14:37:50.741 SingleUserNotebookApp notebookapp:1978] received signal 15, stopping [I 2021-06-12 14:37:51.022 SingleUserNotebookApp notebookapp:2145] Shutting down 0 kernels [I 2021-06-12 14:37:51.022 SingleUserNotebookApp notebookapp:2160] Shutting down 0 terminals SIGTERM received jupyterhub-singleuser ended gracefully ``` This is the output from running the `jupyterhub` command on the login node: ``` [I 2021-06-12 17:17:18.610 JupyterHub app:2463] Running JupyterHub version 1.4.1 [I 2021-06-12 17:17:18.623 JupyterHub app:2493] Using Authenticator: jupyter_saml2authenticator.auth.Saml2Authenticator [I 2021-06-12 17:17:18.623 JupyterHub app:2493] Using Spawner: wrapspawner.wrapspawner.ProfilesSpawner [I 2021-06-12 17:17:18.623 JupyterHub app:2493] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.4.1 [I 2021-06-12 17:17:18.967 JupyterHub app:1534] Loading cookie_secret from /home/iv/jupyterhub/jupyterhub_cookie_secret [I 2021-06-12 17:17:19.326 JupyterHub proxy:497] Generating new CONFIGPROXY_AUTH_TOKEN [W 2021-06-12 17:17:19.328 JupyterHub app:1808] No admin users, admin interface will be unavailable. [W 2021-06-12 17:17:19.328 JupyterHub app:1809] Add any administrative users to `c.Authenticator.admin_users` in config. [I 2021-06-12 17:17:19.328 JupyterHub app:1838] Not using allowed_users. Any authenticated user will be allowed. [I 2021-06-12 17:17:19.451 JupyterHub app:2530] Initialized 0 spawners in 0.006 seconds [I 2021-06-12 17:17:19.461 JupyterHub proxy:703] Starting proxy @ https://jupyterhub-test.contoso.com:8000/ 17:17:25.092 [ConfigProxy] info: Proxying https://jupyterhub-test.contoso.com:8000 to (no default) 17:17:25.187 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes 17:17:25.679 [ConfigProxy] info: 200 GET /api/routes [I 2021-06-12 17:17:25.712 JupyterHub app:2778] Hub API listening on http://jupyterhub-test.contoso.com:8081/hub/ 17:17:25.714 [ConfigProxy] info: 200 GET /api/routes [I 2021-06-12 17:17:25.714 JupyterHub proxy:347] Checking routes [I 2021-06-12 17:17:25.714 JupyterHub proxy:432] Adding route for Hub: / => http://jupyterhub-test.contoso.com:8081 17:17:25.717 [ConfigProxy] info: Adding route / -> http://jupyterhub-test.contoso.com:8081 17:17:25.718 [ConfigProxy] info: Route added / -> http://jupyterhub-test.contoso.com:8081 17:17:25.719 [ConfigProxy] info: 201 POST /api/routes/ [I 2021-06-12 17:17:25.720 JupyterHub app:2853] JupyterHub is now running at https://jupyterhub-test.contoso.com:8000/ [I 2021-06-12 17:17:56.716 JupyterHub log:189] 302 GET / -> /hub/ (@aaa.aaa.aaa.aaa) 43.69ms [I 2021-06-12 17:17:56.951 JupyterHub log:189] 302 GET /hub/ -> /hub/spawn (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 122.05ms [I 2021-06-12 17:17:57.202 JupyterHub log:189] 200 GET /hub/spawn (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 140.49ms [I 2021-06-12 17:17:58.929 JupyterHub provider:574] Creating oauth client jupyterhub-user-aerobinsoniv%40contoso.com [I 2021-06-12 17:17:59.148 JupyterHub batchspawner:262] Spawner submitting job using sudo -E -u aerobinsoniv@contoso.com sbatch --parsable [I 2021-06-12 17:17:59.148 JupyterHub batchspawner:263] Spawner submitted script: #!/bin/bash #SBATCH --output=/home/aerobinsoniv@admin.contoso.com/jupyterhub_slurmspawner_%j.log #SBATCH --job-name=spawner-jupyterhub #SBATCH --chdir=/home/aerobinsoniv@admin.contoso.com #SBATCH --export=PATH,CONDA_DEFAULT_ENV,LANG,JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,USER,HOME,SHELL #SBATCH --get-user-env=L #SBATCH --time=6:00:00 #SBATCH --mem=2gb #SBATCH --cpus-per-task=2 set -euo pipefail trap 'echo SIGTERM received' TERM which jupyterhub-singleuser srun batchspawner-singleuser jupyterhub-singleuser --ip=0.0.0.0 echo "jupyterhub-singleuser ended gracefully" [I 2021-06-12 17:17:59.419 JupyterHub batchspawner:266] Job submitted. cmd: sudo -E -u aerobinsoniv@contoso.com sbatch --parsable output: 188 [I 2021-06-12 17:17:59.874 JupyterHub log:189] 302 POST /hub/spawn -> /hub/spawn-pending/aerobinsoniv@contoso.com (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 1009.44ms [I 2021-06-12 17:18:00.049 JupyterHub pages:402] aerobinsoniv@contoso.com is pending spawn [I 2021-06-12 17:18:00.061 JupyterHub log:189] 200 GET /hub/spawn-pending/aerobinsoniv@contoso.com (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 32.47ms [I 2021-06-12 17:18:03.978 JupyterHub api:23] key is port, value 41166 [I 2021-06-12 17:18:03.982 JupyterHub api:24] hasattr(spawner, key) is True [I 2021-06-12 17:18:03.986 JupyterHub log:189] 201 POST /hub/api/batchspawner (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 40.37ms [I 2021-06-12 17:18:04.078 JupyterHub batchspawner:420] Notebook server job 188 started at jupyterhub-c5:41166 [I 2021-06-12 17:18:08.636 JupyterHub base:909] User aerobinsoniv@contoso.com took 9.766 seconds to start [I 2021-06-12 17:18:08.639 JupyterHub proxy:285] Adding user aerobinsoniv@contoso.com to proxy /user/aerobinsoniv@contoso.com/ => http://jupyterhub-c5:41166 [I 2021-06-12 17:18:08.643 JupyterHub log:189] 200 GET /hub/api (@aaa.aaa.aaa.aaa) 1.39ms 17:18:08.668 [ConfigProxy] info: Adding route /user/aerobinsoniv@contoso.com -> http://jupyterhub-c5:41166 17:18:08.669 [ConfigProxy] info: Route added /user/aerobinsoniv@contoso.com -> http://jupyterhub-c5:41166 17:18:08.670 [ConfigProxy] info: 201 POST /api/routes/user/aerobinsoniv@contoso.com [I 2021-06-12 17:18:08.718 JupyterHub log:189] 200 POST /hub/api/users/aerobinsoniv@contoso.com/activity (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 47.25ms [I 2021-06-12 17:18:08.719 JupyterHub users:677] Server aerobinsoniv@contoso.com is ready [I 2021-06-12 17:18:08.720 JupyterHub log:189] 200 GET /hub/api/users/aerobinsoniv@contoso.com/server/progress (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 8398.65ms [I 2021-06-12 17:18:08.880 JupyterHub log:189] 302 GET /hub/spawn-pending/aerobinsoniv@contoso.com -> /user/aerobinsoniv@contoso.com/ (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 13.14ms [I 2021-06-12 17:18:09.415 JupyterHub log:189] 302 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb -> /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@aaa.aaa.aaa.aaa) 1.53ms [I 2021-06-12 17:18:09.418 JupyterHub log:189] 302 GET /static/style/style.min.css?v=56dfd556850eb17b7998c6828467598a322b41593edc758739c66cb2c3fea98f23d0dd8bf8b9b0f5d67bb976a50e4c34f789fe640cbb440fa089e1bf5ec170bd -> /hub/static/style/style.min.css?v=56dfd556850eb17b7998c6828467598a322b41593edc758739c66cb2c3fea98f23d0dd8bf8b9b0f5d67bb976a50e4c34f789fe640cbb440fa089e1bf5ec170bd (@aaa.aaa.aaa.aaa) 0.93ms [I 2021-06-12 17:18:09.422 JupyterHub log:189] 302 GET /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a -> /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@aaa.aaa.aaa.aaa) 0.95ms [I 2021-06-12 17:18:09.431 JupyterHub log:189] 302 GET /custom/custom.css -> /hub/custom/custom.css (@aaa.aaa.aaa.aaa) 1.63ms [I 2021-06-12 17:18:09.432 JupyterHub log:189] 302 GET /static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 -> /hub/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@aaa.aaa.aaa.aaa) 1.63ms [I 2021-06-12 17:18:09.435 JupyterHub log:189] 302 GET /static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 0.93ms [I 2021-06-12 17:18:09.531 JupyterHub log:189] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 0.99ms [I 2021-06-12 17:18:09.545 JupyterHub log:189] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 2.38ms [I 2021-06-12 17:18:09.546 JupyterHub log:189] 302 GET /static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 -> /hub/static/components/requirejs/require.js?v=d37b48bb2137faa0ab98157e240c084dd5b1b5e74911723aa1d1f04c928c2a03dedf922d049e4815f7e5a369faa2e6b6a1000aae958b7953b5cc60411154f593 (@aaa.aaa.aaa.aaa) 2.45ms [W 2021-06-12 17:18:09.548 JupyterHub log:189] 404 GET /hub/static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=fb45616eef2c454960f91fcd2a04efeda84cfacccf0c5d741ba2793dc1dbd6d3ab01aaae6485222945774c7d7a9a2e9fb87e0d8ef1ea96893aa6906147a371bb (@aaa.aaa.aaa.aaa) 3.09ms [W 2021-06-12 17:18:09.637 JupyterHub log:189] 404 GET /hub/static/style/style.min.css?v=56dfd556850eb17b7998c6828467598a322b41593edc758739c66cb2c3fea98f23d0dd8bf8b9b0f5d67bb976a50e4c34f789fe640cbb440fa089e1bf5ec170bd (@aaa.aaa.aaa.aaa) 1.63ms [W 2021-06-12 17:18:09.640 JupyterHub log:189] 404 GET /hub/static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=5edf53bf6bb9c3b1ddafd8594825a7e2ed621f19423e569c985162742f63911c09eba2c529f8fb47aebf27fafdfe287d563347f58c1126b278189a18871b6a9a (@aaa.aaa.aaa.aaa) 1.39ms [W 2021-06-12 17:18:09.729 JupyterHub log:189] 404 GET /hub/custom/custom.css (aerobinsoniv@contoso.com@aaa.aaa.aaa.aaa) 57.21ms [W 2021-06-12 17:18:09.733 JupyterHub log:189] 404 GET /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 2.69ms [W 2021-06-12 17:18:09.734 JupyterHub log:189] 404 GET /hub/static/components/es6-promise/promise.min.js?v=bea335d74136a63ae1b5130f5ac9a50c6256a5f435e6e09fef599491a84d834a8b0f011ca3eaaca3b4ab6a2da2d3e1191567a2f171e60da1d10e5b9d52f84184 (@aaa.aaa.aaa.aaa) 3.16ms [W 2021-06-12 17:18:09.735 JupyterHub log:189] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 3.33ms [W 2021-06-12 17:18:09.779 JupyterHub log:189] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 1.23ms [I 2021-06-12 17:18:09.854 JupyterHub log:189] 302 GET /static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 -> /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 1.19ms [W 2021-06-12 17:18:09.961 JupyterHub log:189] 404 GET /hub/static/components/react/react.production.min.js?v=9a0aaf84a316c8bedd6c2ff7d5b5e0a13f8f84ec02442346cba0b842c6c81a6bf6176e64f3675c2ebf357cb5bb048e0b527bd39377c95681d22468da3d5de735 (@aaa.aaa.aaa.aaa) 1.05ms [I 2021-06-12 17:18:10.067 JupyterHub log:189] 302 GET /static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 -> /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 1.01ms [W 2021-06-12 17:18:10.171 JupyterHub log:189] 404 GET /hub/static/components/react/react-dom.production.min.js?v=6fc58c1c4736868ff84f57bd8b85f2bdb985993a9392718f3b4af4bfa10fb4efba2b4ddd68644bd2a8daf0619a3844944c9c43f8528364a1aa6fc01ec1b8ae84 (@aaa.aaa.aaa.aaa) 1.02ms [I 2021-06-12 17:18:10.278 JupyterHub log:189] 302 GET /static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec -> /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 0.99ms [W 2021-06-12 17:18:10.383 JupyterHub log:189] 404 GET /hub/static/components/create-react-class/index.js?v=894ad57246e682b4cfbe7cd5e408dcd6b38d06af4de4f3425991e2676fdc2ef1732cbd19903104198878ae77de12a1996de3e7da3a467fb226bdda8f4618faec (@aaa.aaa.aaa.aaa) 0.97ms [I 2021-06-12 17:18:10.521 JupyterHub log:189] 302 GET /static/components/jquery/jquery.min.js?v=20210612171805 -> /hub/static/components/jquery/jquery.min.js?v=20210612171805 (@aaa.aaa.aaa.aaa) 0.95ms [W 2021-06-12 17:18:10.625 JupyterHub log:189] 404 GET /hub/static/components/jquery/jquery.min.js?v=20210612171805 (@aaa.aaa.aaa.aaa) 1.41ms ^C[C 2021-06-12 17:18:15.594 JupyterHub app:2941] Received signal SIGINT, initiating shutdown... [I 2021-06-12 17:18:15.595 JupyterHub app:2582] Cleaning up single-user servers... [I 2021-06-12 17:18:15.595 JupyterHub proxy:772] Cleaning up proxy[4430]... 17:18:15.607 [ConfigProxy] warn: Terminated [I 2021-06-12 17:18:15.809 JupyterHub batchspawner:431] Stopping server job 188 [I 2021-06-12 17:18:15.828 JupyterHub batchspawner:316] Cancelling job 188: sudo -E -u aerobinsoniv@contoso.com scancel 188 [I 2021-06-12 17:18:18.195 JupyterHub app:2614] ...done ```

Jupyterhub.har.zip

welcome[bot] commented 3 years ago

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

jbaksta commented 3 years ago

I find that usually the 404 on the spawn has something to do with a missing environment variable that describes one of the communication paths/routes. We've got some add-on scripts that attempt to verify that the variables are set and if not, set them. The following are ones that I've made sure are accessible in our environment to work right (and this can be a little tricky as every once in a while, we had some environment sanitization for various reasons. 

JUPYTERHUB_API_URL
JUPYTERHUB_BASE_URL
JUPYTERHUB_CLIENT_ID
JUPYTERHUB_OAUTH_CALLBACK_URL
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_SERVICE_PREFIX
JUPYTERHUB_USER
ivhacks commented 3 years ago

Hi @jbaksta , thanks for your response, this looks very promising! How did you figure out what values to use in your setup?

jbaksta commented 3 years ago

Well, I had a test cluster set up w/ Docker basically that was extremely vanilla Slurm installation when I did this. Then I just started comparing the outputs of the environment already knowing that we didn't allow many environment variables to be passed to the jobs because of OS differences (SLES submission host, CentOS batch nodes). So I hacked on our batch_script to eventually make it look like my vanilla job instance. Most of the variables rely on some notion of your site setup and the jupyterhub API routes I guess. Some of them are just proper combinations of the others; some are something you can probably regenerate from your jupyterhub_config.py

ivhacks commented 3 years ago

I just logged into the working production server and looked at the environment variables. Hardcoding JUPYTERHUB_SERVICE_PREFIX to "/user/[my email]" on line 805 of spawner.py (for the benefit of future readers) caused it to start working. Obviously this isn't a final fix, but now that the cause of the issue has been isolated I can solve it. Thank you so much, I have been trying to figure this out for months!!

jbaksta commented 3 years ago

Under normal circumstances, I'd think it should work without modifications. But often times, this ends up being something missing in the keepvars or something I think you can add to the keepvars_extra. We override our batch_script elements quite heavily and based on specific clusters that are being submitted to and that's when you find things missing depending on how the batch system is configured. Regardless, happy to see that gets you farther.

ivhacks commented 3 years ago

Alas, for software like this meant for HPC setups with significant user configuration expected, "normal circumstances" seem to be mostly theoretical.... Anyway, thanks for the additional info.

Since it's unclear if this is a real bug or just user error, I'll leave it up to you and the other Batchspawner devs whether to close this issue.

basvandervlies commented 3 years ago

I just installed Jupyterhub 1.4.1, batchspawner 1.1.0 and wrapspawner and encounterd the same http error. I had to set:

if [[ -z ${JUPYTERHUB_SERVICE_PREFIX} ]]
then
    export JUPYTERHUB_SERVICE_PREFIX=${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}
fi

I our previous environment with HUB 1.1.0, and batchspawner/wrapspawner installed from git I did not encounter this problems

ivhacks commented 3 years ago

That's the exact same thing I did as my final fix, just in Bash instead of Python. So maybe it isn't just me. @basvandervlies are you willing to provide more information about your setup?

basvandervlies commented 3 years ago

@aerobinsonIV of course. We have HPC cluster we run multiple jupyterhub setups for course/training and production. For each jupyterhub setup we can specify which version/software must be used and we generate the config files based on templates and json data. For this we have written a services framework:

This framework will generate the apache, systemd unit files , jupyterhub start script and the jupyterhub configuration file.

I am now testing the jupytherhub 14.1 version with batchspawner and wrapspawner and encountered that it would not start . That is fixed by the patch from @jbaksta. After that we got 404 errors. So I added the above bash statement to the batch script that is defined in the jupyterhub config.

c.SlurmSpawner.batch_script = '''#!/bin/bash
#SBATCH -o "jupyterhub-2021-%j.out"
#SBATCH -e "jupyterhub-2021-%j.error"
#SBATCH --job-name=jupyterhub-2021-singleuser
#SBATCH -D {{homedir}}
#SBATCH --get-user-env=L
{% if partition  %}#SBATCH --partition={{partition}}
{% endif %}{% if runtime    %}#SBATCH --time={{runtime}}
{% endif %}{% if memory     %}#SBATCH --mem={{memory}}
{% endif %}{% if nprocs     %}#SBATCH --cpus-per-task={{nprocs}}
{% endif %}{% if reservation%}#SBATCH --reservation={{reservation}}
{% endif %}{% if options    %}#SBATCH {{options}}{% endif %}

trap 'echo SIGTERM received' TERM

## Set the User writeable jupyter notebook config directory
#
export JUPYTER_CONFIG_DIR=${HOME}/.jupyter-2021

## Always clean the module environment
#
module purge
if [[ -f ${JUPYTER_CONFIG_DIR}/env ]]
then
    source ${JUPYTER_CONFIG_DIR}/env
else
    {{prologue}}
fi

## Some batch settings before starting the notebook
#
export LMOD_CMD=/hpc/eb/modules-4.0.0/libexec/modulecmd.tcl

if [[ -x ${JUPYTER_CONFIG_DIR}/script ]]
then
    ${JUPYTER_CONFIG_DIR}/script
fi

## jupyter notebook extensions enable/disable/remove
jupyter nbextension enable --py jupyterlmod --user

## jupyter server extensions enable/disable
#
jupyter serverextension enable --py jupyterlmod --user
jupyter serverextension enable jupyter_server_proxy --user

if [[ -z ${JUPYTERHUB_SERVICE_PREFIX} ]]
then
    export JUPYTERHUB_SERVICE_PREFIX=${JUPYTERHUB_BASE_URL}user/${JUPYTERHUB_USER}
fi
echo "JUPYTERHUB_API_URL = ${JUPYTERHUB_API_URL}"
echo "JUPYTERHUB_BASE_URL = ${JUPYTERHUB_BASE_URL}"
echo "JUPYTERHUB_CLIENT_ID = ${JUPYTERHUB_CLIENT_ID}"
echo "JUPYTERHUB_OAUTH_CALLBACK_URL = ${JUPYTERHUB_OAUTH_CALLBACK_URL}"
echo "JUPYTERHUB_SERVER_NAME = ${JUPYTERHUB_SERVER_NAME}"
echo "JUPYTERHUB_SERVICE_PREFIX = ${JUPYTERHUB_SERVICE_PREFIX}"
echo "JUPYTERHUB_USER = ${JUPYTERHUB_USER}"

which batchspawner-singleuser
which jupyterhub-singleuser
#{% if srun %}{{srun}} {% endif %} {{cmd}}
# {{cmd}}   --NotebookApp.terminado_settings="{'shell_command': ['bash']}" --notebook-dir=${notebook_dir}
{{cmd}}   --NotebookApp.terminado_settings="{'shell_command': ['bash']}"

echo "jupyterhub-singleuser ended gracefully"
{{epilogue}}