spawned notebook servers unable to determine the jupyterhub user

johannjc commented 3 weeks ago

I've recently attempted to upgrade our jupyterhub server to a more recent version and am running into an issue. The spawning piece works fine (using slurm) - and the notebook server connects back to the hub and the hub then redirects to the server - but the notebook server is not able to determine the user and displays the message "A Jupyter Server is Running" in the browser. From trying to debug this myself - it looks like the _get_user method in jupyterhub/services/auth.py is failing to determine the hub user associated with the notebook server.

Here is the output from the notebook server

[D 2024-10-29 16:29:17.937 ServerApp] No files in /home/jcarrol5/.ipython/nbextensions
[D 2024-10-29 16:29:17.968 ServerApp] Searching ['/sfw/rhel9-x86_64/jupyter/b1/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2024-10-29 16:29:17.968 ServerApp] Looking for jupyter_config in /etc/jupyter
[D 2024-10-29 16:29:17.968 ServerApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2024-10-29 16:29:17.968 ServerApp] Looking for jupyter_config in /sfw/rhel9-x86_64/jupyter/b1/etc/jupyter
[D 2024-10-29 16:29:17.969 ServerApp] Looking for jupyter_server_config in /etc/jupyter
[D 2024-10-29 16:29:17.969 ServerApp] Looking for jupyter_server_config in /usr/local/etc/jupyter
[D 2024-10-29 16:29:17.969 ServerApp] Looking for jupyter_server_config in /sfw/rhel9-x86_64/jupyter/b1/etc/jupyter
[D 2024-10-29 16:29:17.990 ServerApp] Paths used for configuration of jupyter_server_config:
        /etc/jupyter/jupyter_server_config.json
[D 2024-10-29 16:29:17.990 ServerApp] Paths used for configuration of jupyter_server_config:
        /usr/local/etc/jupyter/jupyter_server_config.json
[D 2024-10-29 16:29:17.991 ServerApp] Paths used for configuration of jupyter_server_config:
        /sfw/rhel9-x86_64/jupyter/b1/etc/jupyter/jupyter_server_config.d/jupyter_server_terminals.json
        /sfw/rhel9-x86_64/jupyter/b1/etc/jupyter/jupyter_server_config.json
[D 2024-10-29 16:29:18.020 ServerApp] Extension package jupyterhub took 0.0002s to import
[I 2024-10-29 16:29:18.152 ServerApp] Extension package jupyter_server_terminals took 0.1313s to import
[D 2024-10-29 16:29:18.158 TerminalsExtensionApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue {'update': {'jupyter_server_terminals': True}}>}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[D 2024-10-29 16:29:18.158 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_server_terminals': True}>}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[I 2024-10-29 16:29:18.159 ServerApp] jupyter_server_terminals | extension was successfully linked.
[I 2024-10-29 16:29:18.159 JupyterHubSingleUser] Starting jupyterhub single-user server extension version 5.2.1
[W 2024-10-29 16:29:18.160 JupyterHubSingleUser] No default url found in config or known extensions, searching other extensions for default_url
[W 2024-10-29 16:29:18.160 JupyterHubSingleUser] Found no extension with a default URL, UI will likely be unavailable
[D 2024-10-29 16:29:18.161 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_server_terminals': True}>}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[D 2024-10-29 16:29:18.164 JupyterHubSingleUser] Config changed: {'ServerApp': {'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 42261, 'ip': '0.0.0.0', 'base_url': '/user/jcarrol5/', 'default_url': '/', 'keyfile': '', 'certfile': '', 'client_ca': '', 'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_server_terminals': True}>}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/gpfs/fs1/sfw3/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyterhub/singleuser/templates']}>}, 'ExtensionApp': {'log_level': 'DEBUG'}}
[D 2024-10-29 16:29:18.164 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_server_terminals': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 42261, 'ip': '0.0.0.0', 'base_url': '/user/jcarrol5/', 'default_url': '/', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'FileContentsManager': {'delete_to_trash': False}}
[I 2024-10-29 16:29:18.165 ServerApp] jupyterhub | extension was successfully linked.
[D 2024-10-29 16:29:18.166 ServerApp] Config changed: {'ServerApp': {'log_level': 'DEBUG', 'jpserver_extensions': <LazyConfigValue value={'jupyterhub': True, 'jupyter_server_terminals': True}>, 'identity_provider_class': <class 'jupyterhub.singleuser.extension.JupyterHubIdentityProvider'>, 'allow_remote_access': True, 'open_browser': False, 'trust_xheaders': True, 'quit_button': False, 'port_retries': 0, 'answer_yes': True, 'port': 42261, 'ip': '0.0.0.0', 'base_url': '/user/jcarrol5/', 'default_url': '/', 'keyfile': '', 'certfile': '', 'client_ca': ''}, 'ExtensionApp': {'log_level': 'DEBUG'}, 'FileContentsManager': {'delete_to_trash': False}, 'NotebookApp': {'extra_template_paths': <LazyConfigValue {'extend': ['/gpfs/fs1/sfw3/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyterhub/singleuser/templates']}>}}
[I 2024-10-29 16:29:18.323 ServerApp] jupyter_server_terminals | extension was successfully loaded.
[D 2024-10-29 16:29:18.349 JupyterHubSingleUser] jupyterhub version 5.2.0 != jupyterhub-singleuser version 5.2.1.
[I 2024-10-29 16:29:18.349 JupyterHubSingleUser] Updating Hub with activity every 300 seconds
[I 2024-10-29 16:29:18.350 ServerApp] jupyterhub | extension was successfully loaded.
[I 2024-10-29 16:29:18.357 ServerApp] Serving notebooks from local directory: /gpfs/fs1/home/jcarrol5
[I 2024-10-29 16:29:18.357 ServerApp] Jupyter Server 2.14.2 is running at:
[I 2024-10-29 16:29:18.357 ServerApp] http://bhg0065:42261/user/jcarrol5/?token=...
[I 2024-10-29 16:29:18.357 ServerApp]     http://127.0.0.1:42261/user/jcarrol5/?token=...
[I 2024-10-29 16:29:18.357 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[D 2024-10-29 16:29:18.378 JupyterHubSingleUser] Notifying Hub of activity 2024-10-29T20:29:18.229232Z
[D 2024-10-29 16:29:19.080 ServerApp] No user identified
[I 2024-10-29 16:29:19.081 ServerApp] 200 GET /user/jcarrol5/api (@192.168.31.75) 10.33ms
[D 2024-10-29 16:29:19.121 ServerApp] No user identified
[D 2024-10-29 16:29:19.121 ServerApp] Using contents: services/contents
[D 2024-10-29 16:29:19.142 ServerApp] Path favicon.ico served from /sfw/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyter_server/static/favicon.ico
[D 2024-10-29 16:29:19.166 ServerApp] Path style/bootstrap.min.css served from /sfw/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyter_server/static/style/bootstrap.min.css
[D 2024-10-29 16:29:19.179 ServerApp] Path style/bootstrap-theme.min.css served from /sfw/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyter_server/static/style/bootstrap-theme.min.css
[D 2024-10-29 16:29:19.190 ServerApp] Path style/index.css served from /sfw/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyter_server/static/style/index.css
[D 2024-10-29 16:29:19.191 ServerApp] Path logo/logo.png served from /sfw/rhel9-x86_64/jupyter/b1/lib/python3.9/site-packages/jupyter_server/static/logo/logo.png
[D 2024-10-29 16:29:19.209 ServerApp] 304 GET /user/jcarrol5/ (@128.151.18.227) 88.89ms

and jupyterhub

[jupyterhub@bh3-jupyter ~]$ ./runjupyter.sh
[W 2024-10-29 18:38:03.157 JupyterHub app:964] JupyterHub.proxy_auth_token is deprecated in JupyterHub 0.8, use ConfigurableHTTPProxy.auth_token
[I 2024-10-29 18:38:03.157 JupyterHub app:3352] Running JupyterHub version 5.2.0
[I 2024-10-29 18:38:03.157 JupyterHub app:3382] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuthenticator-1.3.2
[I 2024-10-29 18:38:03.157 JupyterHub app:3382] Using Spawner: wrapspawner.wrapspawner.DropDownOptionsSpawner
[I 2024-10-29 18:38:03.158 JupyterHub app:3382] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.2.0
[D 2024-10-29 18:38:03.160 JupyterHub app:3312] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2024-10-29 18:38:03.160 JupyterHub app:1837] Loading cookie_secret from /opt/jupyterhub/jupyterhub_cookie_secret
[D 2024-10-29 18:38:03.160 JupyterHub app:2004] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2024-10-29 18:38:03.173 JupyterHub orm:1509] database schema version found: 4621fec11365
[D 2024-10-29 18:38:03.223 JupyterHub app:2344] Loading roles into database
[D 2024-10-29 18:38:03.243 JupyterHub app:2691] Purging expired APITokens
[D 2024-10-29 18:38:03.245 JupyterHub app:2691] Purging expired OAuthCodes
[D 2024-10-29 18:38:03.247 JupyterHub app:2691] Purging expired Shares
[D 2024-10-29 18:38:03.248 JupyterHub app:2691] Purging expired ShareCodes
[D 2024-10-29 18:38:03.249 JupyterHub app:2465] Loading role assignments from config
[D 2024-10-29 18:38:03.250 JupyterHub app:2576] Assigning allowed_users to the user role
[D 2024-10-29 18:38:03.254 JupyterHub app:2584] Assigning 0 allowed_users to the user role
[D 2024-10-29 18:38:03.267 JupyterHub app:2976] Initializing spawners
[D 2024-10-29 18:38:03.274 JupyterHub app:3126] Loaded users:

[I 2024-10-29 18:38:03.274 JupyterHub app:3422] Initialized 0 spawners in 0.007 seconds
[I 2024-10-29 18:38:03.277 JupyterHub metrics:373] Found 1 active users in the last ActiveUserPeriods.twenty_four_hours
[I 2024-10-29 18:38:03.277 JupyterHub metrics:373] Found 1 active users in the last ActiveUserPeriods.seven_days
[I 2024-10-29 18:38:03.278 JupyterHub metrics:373] Found 1 active users in the last ActiveUserPeriods.thirty_days
[W 2024-10-29 18:38:03.278 JupyterHub proxy:748] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2024-10-29 18:38:03.278 JupyterHub proxy:752] Starting proxy @ http://:8000
[D 2024-10-29 18:38:03.278 JupyterHub proxy:753] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://192.168.223.76:8081/hub/error', '--log-level', 'info']
[D 2024-10-29 18:38:03.281 JupyterHub proxy:670] Writing proxy pid file: jupyterhub-proxy.pid
[D 2024-10-29 18:38:03.281 JupyterHub utils:272] Waiting 10s for server at bh3-jupyter:8000
[D 2024-10-29 18:38:03.282 JupyterHub utils:119] Server at bh3-jupyter:8000 not ready: [Errno 111] Connection refused
[D 2024-10-29 18:38:03.282 JupyterHub utils:272] Waiting 10s for server at 127.0.0.1:8001
[D 2024-10-29 18:38:03.282 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-10-29 18:38:03.409 JupyterHub utils:119] Server at 127.0.0.1:8001 not ready: [Errno 111] Connection refused
[D 2024-10-29 18:38:03.420 JupyterHub utils:119] Server at bh3-jupyter:8000 not ready: [Errno 111] Connection refused
18:38:03.511 [ConfigProxy] info: Proxying http://*:8000 to (no default)
18:38:03.513 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2024-10-29 18:38:03.644 JupyterHub utils:280] Server at 127.0.0.1:8001 responded in 0.36s
[D 2024-10-29 18:38:03.737 JupyterHub utils:280] Server at bh3-jupyter:8000 responded in 0.46s
[D 2024-10-29 18:38:03.737 JupyterHub proxy:832] Proxy started and appears to be up
[D 2024-10-29 18:38:03.739 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2024-10-29 18:38:03.745 JupyterHub app:3738] Hub API listening on http://192.168.223.76:8081/hub/
[D 2024-10-29 18:38:03.746 JupyterHub proxy:389] Fetching routes to check
[D 2024-10-29 18:38:03.746 JupyterHub proxy:925] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
18:38:03.746 [ConfigProxy] info: 200 GET /api/routes
18:38:03.747 [ConfigProxy] info: 200 GET /api/routes
[D 2024-10-29 18:38:03.747 JupyterHub proxy:392] Checking routes
[I 2024-10-29 18:38:03.747 JupyterHub proxy:477] Adding route for Hub: / => http://192.168.223.76:8081
[D 2024-10-29 18:38:03.747 JupyterHub proxy:925] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
18:38:03.749 [ConfigProxy] info: Adding route / -> http://192.168.223.76:8081
18:38:03.750 [ConfigProxy] info: Route added / -> http://192.168.223.76:8081
18:38:03.751 [ConfigProxy] info: 201 POST /api/routes/
[I 2024-10-29 18:38:03.751 JupyterHub app:3779] JupyterHub is now running at http://:8000
[D 2024-10-29 18:38:03.751 JupyterHub app:3345] It took 0.660 seconds for the Hub to start
[D 2024-10-29 18:38:05.497 JupyterHub base:411] Refreshing auth for jcarrol5
[D 2024-10-29 18:38:05.497 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/jcarrol5 via scope servers!server=jcarrol5/
[D 2024-10-29 18:38:05.497 JupyterHub user:496] Creating <class 'wrapspawner.wrapspawner.DropDownOptionsSpawner'> for jcarrol5:
[W 2024-10-29 18:38:05.498 JupyterHub spawner:179]
    The shared database session at Spawner.db is deprecated, and will be removed.
    Please manage your own database and connections.

    Contact JupyterHub at https://github.com/jupyterhub/jupyterhub/issues/3700
    if you have questions or ideas about direct database needs for your Spawner.

[I 2024-10-29 18:38:05.524 JupyterHub log:192] 200 GET /hub/spawn-pending/jcarrol5?_xsrf=[secret] (jcarrol5@128.151.18.227) 42.11ms
[D 2024-10-29 18:38:05.667 JupyterHub log:192] 200 GET /hub/static/js/not_running.js?v=20241029183803 (@128.151.18.227) 3.19ms
[D 2024-10-29 18:38:05.697 JupyterHub log:192] 200 GET /hub/static/js/utils.js?v=20241029183803 (@128.151.18.227) 0.59ms
[D 2024-10-29 18:38:06.764 JupyterHub scopes:1010] Checking access to /hub/spawn/jcarrol5 via scope servers!server=jcarrol5/
[D 2024-10-29 18:38:06.764 JupyterHub pages:208] Serving options form for jcarrol5
[I 2024-10-29 18:38:06.769 JupyterHub log:192] 200 GET /hub/spawn/jcarrol5 (jcarrol5@128.151.18.227) 8.28ms
[D 2024-10-29 18:38:09.601 JupyterHub scopes:1010] Checking access to /hub/spawn/jcarrol5 via scope servers!server=jcarrol5/
[D 2024-10-29 18:38:09.602 JupyterHub pages:256] Triggering spawn with supplied form options for jcarrol5
[D 2024-10-29 18:38:09.602 JupyterHub base:1095] Initiating spawn for jcarrol5
[D 2024-10-29 18:38:09.602 JupyterHub base:1099] 0/100 concurrent spawns
[D 2024-10-29 18:38:09.602 JupyterHub base:1104] 0 active servers
[I 2024-10-29 18:38:09.623 JupyterHub provider:661] Creating oauth client jupyterhub-user-jcarrol5
[D 2024-10-29 18:38:09.637 JupyterHub user:913] Calling Spawner.start for jcarrol5
[I 2024-10-29 18:38:09.640 JupyterHub batchspawner:281] Spawner script options: {'account': '', 'cluster': '', 'epilogue': '', 'gres': '', 'homedir': '/home/jcarrol5', 'host': '', 'keepvars': 'JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABL
ED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SERVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SE
RVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,JUPYTERHUB_DEBUG,JUPYTERHUB_DISABLE_USER_CONFIG,JUPYTER_ENABLE_LAB,USER,HOME,SHELL', 'keepvars_extra': '', 'memory': '6', 'ngpus': '', 'nprocs': '1', 'options': '-p h100 -q h100', 'partition': 'interactive', 'prologue': '', 'q
os': '', 'queue': 'interactive', 'reservation': '', 'runtime': '0-8:0:00', 'srun': 'srun', 'username': 'jcarrol5', 'cmd': 'batchspawner-singleuser jupyterhub-singleuser', 'cpus': '1', 'days': '0', 'hours': '8', 'minutes': '0'}
[I 2024-10-29 18:38:09.640 JupyterHub batchspawner:282] Spawner submitting command: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/sbatch --parsable
[D 2024-10-29 18:38:09.640 JupyterHub batchspawner:283] Spawner submitting script:
    #!/bin/bash
    #SBATCH --partition=interactive
    #SBATCH --cpus-per-task=1
    #SBATCH --time=0-8:0:00
    #SBATCH -o /scratch/jcarrol5/tmp/jupyterhub_slurmspawner_%j.log
    #SBATCH --job-name=spawner-jupyterhub
    #SBATCH -D /home/jcarrol5
    #SBATCH --mem=6
    #SBATCH --export=JUPYTERHUB_API_TOKEN,JPY_API_TOKEN,JUPYTERHUB_CLIENT_ID,JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED,JUPYTERHUB_HOST,JUPYTERHUB_OAUTH_CALLBACK_URL,JUPYTERHUB_OAUTH_SCOPES,JUPYTERHUB_OAUTH_ACCESS_SCOPES,JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES,JUPYTERHUB_USER,JUPYTERHUB_SE
RVER_NAME,JUPYTERHUB_API_URL,JUPYTERHUB_ACTIVITY_URL,JUPYTERHUB_BASE_URL,JUPYTERHUB_SERVICE_PREFIX,JUPYTERHUB_SERVICE_URL,JUPYTERHUB_PUBLIC_URL,JUPYTERHUB_PUBLIC_HUB_URL,JUPYTERHUB_DEBUG,JUPYTERHUB_DISABLE_USER_CONFIG,JUPYTER_ENABLE_LAB,USER,HOME,SHELL
    ##SBATCH --uid=jcarrol5
    #SBATCH --get-user-env=L
    #SBATCH -p h100 -q h100
    #. /software/modules/init/bash
    #module purge
    #module load circ
    module load bluehive/3.0
    module load jupyter
    unset XDG_RUNTIME_DIR
    #export HOME=/home/jcarrol5
    #export USER=jcarrol5
    env
    #export JUPYTERHUB_SINGLEUSER_EXTENSION=0
    #export JUPYTERHUB_SERVICE_URL=http://0.0.0.0:8000/user/jcarrol5
    echo "batchspawner-singleuser jupyterhub-singleuser"
    batchspawner-singleuser jupyterhub-singleuser --debug

[D 2024-10-29 18:38:09.640 JupyterHub batchspawner:284] Spawner submitting environment: {'JUPYTERHUB_API_TOKEN': 'a9c559837fb944418f4524504892fa2c', 'JPY_API_TOKEN': 'a9c559837fb944418f4524504892fa2c', 'JUPYTERHUB_CLIENT_ID': 'jupyterhub-user-jcarrol5', 'JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED': '0', 'JUPYTERHUB_HOST': '', 'JUPYTERHUB_OAUTH_CALLBACK_URL': '/user/jcarrol5/oauth_callback', 'JUPYTERHUB_OAUTH_SCOPES': '["access:servers!server=jcarrol5/", "access:servers!user=jcarrol5"]', 'JUPYTERHUB_OAUTH_ACCESS_SCOPES': '["access:servers!server=jcarrol5/", "access:servers!user=jcarrol5"]', 'JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES': '[]', 'JUPYTERHUB_USER': 'jcarrol5', 'JUPYTERHUB_SERVER_NAME': '', 'JUPYTERHUB_API_URL': 'http://192.168.223.76:8081/hub/api', 'JUPYTERHUB_ACTIVITY_URL': 'http://192.168.223.76:8081/hub/api/users/jcarrol5/activity', 'JUPYTERHUB_BASE_URL': '/', 'JUPYTERHUB_SERVICE_PREFIX': '/user/jcarrol5/', 'JUPYTERHUB_SERVICE_URL': 'http://0.0.0.0:0/user/jcarrol5/', 'JUPYTERHUB_PUBLIC_URL': '', 'JUPYTERHUB_PUBLIC_HUB_URL': '', 'JUPYTERHUB_DEBUG': '1', 'JUPYTERHUB_DISABLE_USER_CONFIG': '1', 'JUPYTER_ENABLE_LAB': 'yes', 'USER': 'jcarrol5', 'HOME': '/home/jcarrol5', 'SHELL': '/bin/bash'}
[I 2024-10-29 18:38:09.755 JupyterHub batchspawner:287] Job submitted. output: 265
[D 2024-10-29 18:38:09.756 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:09.784 JupyterHub batchspawner:428] Job 265 still pending
[D 2024-10-29 18:38:10.285 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[W 2024-10-29 18:38:10.603 JupyterHub base:201] Rolling back dirty objects IdentitySet([<Server(:0)>])
[I 2024-10-29 18:38:10.612 JupyterHub log:192] 302 POST /hub/spawn/jcarrol5?_xsrf=[secret] -> /hub/spawn-pending/jcarrol5?_xsrf=[secret] (jcarrol5@128.151.18.227) 1006.22ms
[D 2024-10-29 18:38:10.616 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/jcarrol5 via scope servers!server=jcarrol5/
[I 2024-10-29 18:38:10.618 JupyterHub pages:397] jcarrol5 is pending spawn
[I 2024-10-29 18:38:10.622 JupyterHub log:192] 200 GET /hub/spawn-pending/jcarrol5?_xsrf=[secret] (jcarrol5@128.151.18.227) 8.91ms
[D 2024-10-29 18:38:10.815 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:10.853 JupyterHub scopes:1010] Checking access to /hub/api/users/jcarrol5/server/progress via scope read:servers!server=jcarrol5/
[D 2024-10-29 18:38:11.346 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:11.875 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:12.404 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:12.933 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:13.463 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:13.992 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:14.520 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:15.050 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[D 2024-10-29 18:38:15.257 JupyterHub base:366] Recording first activity for <APIToken('a9c5...', user='jcarrol5', client_id='jupyterhub')>
[I 2024-10-29 18:38:15.264 JupyterHub log:192] 200 POST /hub/api/batchspawner (jcarrol5@192.168.205.65) 10.51ms
[D 2024-10-29 18:38:15.607 JupyterHub batchspawner:314] Spawner querying job: sudo -E -u jcarrol5 /sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j 265 -o "%T %B"
[I 2024-10-29 18:38:15.635 JupyterHub batchspawner:463] Notebook server job 265 started at bhg0065:60583
[D 2024-10-29 18:38:15.641 JupyterHub spawner:1475] Polling subprocess every 30s
[D 2024-10-29 18:38:15.647 JupyterHub utils:292] Waiting 30s for server at http://bhg0065:60583/user/jcarrol5/api
[I 2024-10-29 18:38:15.926 JupyterHub log:192] 200 GET /hub/api (@192.168.205.65) 0.76ms
[D 2024-10-29 18:38:15.960 JupyterHub scopes:1010] Checking access to /hub/api/users/jcarrol5/activity via scope users:activity!user=jcarrol5
[D 2024-10-29 18:38:15.965 JupyterHub users:1006] Activity for user jcarrol5: 2024-10-29T22:38:15.884479Z
[D 2024-10-29 18:38:15.965 JupyterHub users:1024] Activity on server jcarrol5/: 2024-10-29T22:38:15.884479Z
[I 2024-10-29 18:38:15.970 JupyterHub log:192] 200 POST /hub/api/users/jcarrol5/activity (jcarrol5@192.168.205.65) 11.76ms
[D 2024-10-29 18:38:16.166 JupyterHub utils:328] Server at http://bhg0065:60583/user/jcarrol5/api responded in 0.52s
[D 2024-10-29 18:38:16.169 JupyterHub _version:67] jupyterhub version 5.2.0 != jupyterhub-singleuser version 5.2.1.
[I 2024-10-29 18:38:16.169 JupyterHub base:1124] User jcarrol5 took 6.567 seconds to start
[I 2024-10-29 18:38:16.169 JupyterHub proxy:331] Adding user jcarrol5 to proxy /user/jcarrol5/ => http://bhg0065:60583
[D 2024-10-29 18:38:16.169 JupyterHub proxy:925] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/user/jcarrol5
18:38:16.171 [ConfigProxy] info: Adding route /user/jcarrol5 -> http://bhg0065:60583
18:38:16.171 [ConfigProxy] info: Route added /user/jcarrol5 -> http://bhg0065:60583
18:38:16.172 [ConfigProxy] info: 201 POST /api/routes/user/jcarrol5
[I 2024-10-29 18:38:16.172 JupyterHub users:899] Server jcarrol5 is ready
[I 2024-10-29 18:38:16.172 JupyterHub log:192] 200 GET /hub/api/users/jcarrol5/server/progress?_xsrf=[secret] (jcarrol5@128.151.18.227) 5325.11ms
[D 2024-10-29 18:38:16.201 JupyterHub scopes:1010] Checking access to /hub/spawn-pending/jcarrol5 via scope servers!server=jcarrol5/
[I 2024-10-29 18:38:16.201 JupyterHub log:192] 302 GET /hub/spawn-pending/jcarrol5?_xsrf=[secret] -> /user/jcarrol5/ (jcarrol5@128.151.18.227) 4.08ms

and the jupyterhub_config.py contains

c = get_config()  #noqa
c.Application.log_level = 'DEBUG'
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address='ldap-proxy.circ.rochester.edu'
c.LDAPAuthenticator.user_search_base='dc=rochester,dc=edu'
c.LDAPAuthenticator.user_attribute='uid'
c.LDAPAuthenticator.bind_dn_template=['uid={username},ou=people,dc=rochester,dc=edu']
c.LDAPAuthenticator.attributes=['uid']
c.JupyterHub.hub_ip = '192.168.223.76'
c.JupyterHub.hub_connect_ip = '192.168.223.76'
c.JupyterHub.hub_ip = '192.168.223.76'
c.JupyterHub.proxy_auth_token = '***************<redacted>*****************'
import wrapspawner
import batchspawner
c.JupyterHub.spawner_class = 'wrapspawner.DropDownOptionsSpawner'
c.SlurmSpawner.batch_submit_cmd = '/sfw/rhel9-x86_64/slurm/24.05.3/bin/sbatch --parsable'
c.SlurmSpawner.batch_query_cmd = '/sfw/rhel9-x86_64/slurm/24.05.3/bin/squeue_wrapper -h -j {job_id} -o "%T %B"'
c.SlurmSpawner.batch_cancel_cmd = '/sfw/rhel9-x86_64/slurm/24.05.3/bin/scancel {job_id}'
c.SlurmSpawner.batch_script = """#!/bin/bash
#SBATCH --partition={queue}
#SBATCH --cpus-per-task={nprocs}
#SBATCH --time={runtime}
#SBATCH -o /scratch/{username}/tmp/jupyterhub_slurmspawner_%j.log
#SBATCH --job-name=spawner-jupyterhub
#SBATCH -D /home/{username}
#SBATCH --mem={memory}
#SBATCH --export={keepvars}
#SBATCH --get-user-env=L
#SBATCH {options}
module load bluehive/3.0
module load jupyter
unset XDG_RUNTIME_DIR
env
echo "{cmd}"
{cmd} --debug
"""
c.DropDownOptionsSpawner.partitions = ['standard','debug','interactive', 'gpu', 'phi', 'preempt', 'gpu-debug']
c.DropDownOptionsSpawner.default_partition_idx=2
c.DropDownOptionsSpawner.default_days=0
c.DropDownOptionsSpawner.default_hours=8
c.DropDownOptionsSpawner.default_minutes=0
c.DropDownOptionsSpawner.default_mem=6
c.DropDownOptionsSpawner.default_cpus=1
c.Spawner.debug = True
c.Spawner.disable_user_config = True
c.Spawner.env_keep = ['JUPYTERHUB_SINGLEUSER_APP'] #, 'JUPYTERHUB_SERVICE_URL']
c.Spawner.environment = { 'JUPYTER_ENABLE_LAB': 'yes' }
c.Spawner.ip = '127.0.0.1'
c.Authenticator.admin_users = set(['jcarrol5'])
c.Authenticator.allowed_users = set(['jcarrol5'])

I'm using a custom child class of WrapSpawner (DropDownOptionsSpawner) but it does the same thing as ProfileSpawner but with multiple drop down boxes to specify partition, time, etc... instead of a list of profiles.

johannjc commented 3 weeks ago

The environment that the notebook server runs in has the following JUPYTERHUB variables set

JUPYTERHUB_SERVICE_URL=http://0.0.0.0:0/user/jcarrol5/
JUPYTERHUB_API_TOKEN=*****<redacted>*******
JUPYTERHUB_BASE_URL=/
JUPYTERHUB_PUBLIC_URL=
JUPYTERHUB_SERVER_NAME=
JUPYTERHUB_DISABLE_USER_CONFIG=1
JUPYTERHUB_SERVICE_PREFIX=/user/jcarrol5/
JUPYTERHUB_OAUTH_CALLBACK_URL=/user/jcarrol5/oauth_callback
JUPYTERHUB_OAUTH_CLIENT_ALLOWED_SCOPES=[]
JUPYTER_ENABLE_LAB=yes
JUPYTERHUB_OAUTH_ACCESS_SCOPES=["access:servers!server=jcarrol5/", "access:servers!user=jcarrol5"]
JUPYTERHUB_COOKIE_HOST_PREFIX_ENABLED=0
JUPYTERHUB_API_URL=http://192.168.223.76:8081/hub/api
JUPYTERLAB_DIR=/home/jcarrol5/.local/share/jupyter/lab
JUPYTERHUB_CLIENT_ID=jupyterhub-user-jcarrol5
JUPYTERHUB_OAUTH_SCOPES=["access:servers!server=jcarrol5/", "access:servers!user=jcarrol5"]
JUPYTERHUB_PUBLIC_HUB_URL=
JUPYTERHUB_HOST=
JUPYTERHUB_DEBUG=1
JUPYTERHUB_USER=jcarrol5
JUPYTERHUB_ACTIVITY_URL=http://192.168.223.76:8081/hub/api/users/jcarrol5/activity

mahendrapaipuri commented 3 weeks ago

Hey,

Set c.Spawner.default_url = '/lab' config and ensure JupyterLab is installed in the single user environment and retry.