:bug: [BUG]: Unable to disable JupyterHub behind Twitcher

tlvu commented 5 months ago

Summary

Setting JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" in env.local is unable to disable JupyterHub behind Twitcher. It breaks the spawn of the personal JupyterLab server completely.

Details

Error in docker logs jupyterhub when JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" is set in env.local: Error starting server lvu: Invalid URL '': No scheme supplied. Perhaps you meant https://?

Full error log:

[I 2024-04-04 16:19:18.682 JupyterHub log:191] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn (lvu@10.10.10.10) 89.01ms
[D 2024-04-04 16:19:18.843 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:19:18.843 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:19:18.845 JupyterHub pages:209] Serving options form for lvu
[I 2024-04-04 16:19:18.875 JupyterHub log:191] 200 GET /jupyter/hub/spawn (lvu@10.10.10.10) 41.23ms
[D 2024-04-04 16:19:21.646 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:19:21.647 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:19:21.647 JupyterHub pages:257] Triggering spawn with supplied form options for lvu
[D 2024-04-04 16:19:21.647 JupyterHub base:344] Refreshing auth for lvu
[E 2024-04-04 16:19:21.650 JupyterHub pages:313] Error starting server lvu: Invalid URL '': No scheme supplied. Perhaps you meant https://?
    Traceback (most recent call last):
    NoneType: None

[E 2024-04-04 16:19:21.650 JupyterHub pages:266] Failed to spawn single-user server with form
    Traceback (most recent call last):
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 262, in _post
        return await self._wrap_spawn_single_user(
      File "/usr/local/lib/python3.10/dist-packages/jupyterhub/handlers/pages.py", line 317, in _wrap_spawn_single_user
        raise web.HTTPError(
    tornado.web.HTTPError: HTTP 500: Internal Server Error (Unhandled error starting server lvu)

[I 2024-04-04 16:19:21.654 JupyterHub log:191] 200 POST /jupyter/hub/spawn?_xsrf=[secret] (lvu@10.10.10.10) 10.75ms
[I 2024-04-04 16:20:05.381 JupyterHub log:191] 200 GET /jupyter/hub/login (@172.18.0.1) 7.34ms

Full log when it works (that switch is not set), starting from the exact same line GET /jupyter/hub/ -> /jupyter/hub/spawn:

[I 2024-04-04 16:24:33.887 JupyterHub log:191] 302 GET /jupyter/hub/ -> /jupyter/hub/spawn (lvu@10.10.10.10) 92.05ms
[D 2024-04-04 16:24:33.959 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:24:33.959 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:24:33.960 JupyterHub pages:209] Serving options form for lvu
[I 2024-04-04 16:24:33.974 JupyterHub log:191] 200 GET /jupyter/hub/spawn (lvu@10.10.10.10) 17.31ms
[D 2024-04-04 16:24:36.638 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn via scope servers
[D 2024-04-04 16:24:36.638 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn via servers
[D 2024-04-04 16:24:36.638 JupyterHub pages:257] Triggering spawn with supplied form options for lvu
[D 2024-04-04 16:24:36.639 JupyterHub base:344] Refreshing auth for lvu
[D 2024-04-04 16:24:36.661 JupyterHub base:961] Initiating spawn for lvu
[D 2024-04-04 16:24:36.661 JupyterHub base:965] 0/100 concurrent spawns
[D 2024-04-04 16:24:36.661 JupyterHub base:970] 0 active servers
[I 2024-04-04 16:24:37.012 JupyterHub provider:659] Creating oauth client jupyterhub-user-lvu
[D 2024-04-04 16:24:37.450 JupyterHub user:794] Calling Spawner.start for lvu
[I 2024-04-04 16:24:37.451 JupyterHub dockerspawner:1262] pulling pavics/workflow-tests:py39-230601-1-update240116
[I 2024-04-04 16:24:37.641 JupyterHub log:191] 302 POST /jupyter/hub/spawn?_xsrf=[secret] -> /jupyter/hub/spawn-pending/lvu?_xsrf=[secret] (lvu@10.10.10.10) 1004.86ms
[D 2024-04-04 16:24:37.786 JupyterHub scopes:877] Checking access to /jupyter/hub/spawn-pending/lvu via scope servers
[D 2024-04-04 16:24:37.787 JupyterHub scopes:690] Argument-based access to /jupyter/hub/spawn-pending/lvu via servers
[I 2024-04-04 16:24:37.788 JupyterHub pages:398] lvu is pending spawn
[I 2024-04-04 16:24:37.805 JupyterHub log:191] 200 GET /jupyter/hub/spawn-pending/lvu?_xsrf=[secret] (lvu@10.10.10.10) 23.01ms
[D 2024-04-04 16:24:37.949 JupyterHub dockerspawner:1027] Getting container 'jupyter-lvu'
[I 2024-04-04 16:24:37.952 JupyterHub dockerspawner:1033] Container 'jupyter-lvu' is gone
[D 2024-04-04 16:24:37.959 JupyterHub dockerspawner:1205] Starting host with config: {REMOVED}
[D 2024-04-04 16:24:38.888 JupyterHub scopes:877] Checking access to /jupyter/hub/api/users/lvu/server/progress via scope read:servers
[D 2024-04-04 16:24:38.888 JupyterHub scopes:690] Argument-based access to /jupyter/hub/api/users/lvu/server/progress via read:servers
[I 2024-04-04 16:24:40.576 JupyterHub dockerspawner:1311] Created container jupyter-lvu (id: f68bcca) from image pavics/workflow-tests:py39-230601-1-update240116
[I 2024-04-04 16:24:40.576 JupyterHub dockerspawner:1335] Starting container jupyter-lvu (id: f68bcca)
[D 2024-04-04 16:24:42.706 JupyterHub spawner:1384] Polling subprocess every 30s
[D 2024-04-04 16:24:42.707 JupyterHub dockerspawner:972] Persisting state for lvu: container name=jupyter-lvu, id=f68bcca87532c4d696a28d476337deee7bcde561062012900fe375b1a65c6807

To Reproduce

Steps to reproduce the behavior:

birdhouse-deploy version 2.1.2 (but should be broken since 1.36.0 when JupyterHub was put behind Twitcher)
set export JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" in env.local
login to JupyterHub will work
but spawning the personal server will fail with the log above server-side

Environment

Information	Value
Server/Platform URL
Version Tag/Commit	2.1.2
Related issues/PR	https://github.com/bird-house/birdhouse-deploy/pull/358 (tag 1.36.0)
Related components	jupyterhub, twitcher, magpie
Custom configuration

Concerned Organizations

@mishaschwartz @fmigneault

tlvu commented 5 months ago

@mishaschwartz I merged the pending https://github.com/Ouranosinc/jupyterhub/pull/29 so you are clear to do any fixes in that repo.

mishaschwartz commented 5 months ago

@tlvu

Is JUPYTERHUB_CRYPT_KEY set? If so, then the MagpieAuthenticator.refresh_pre_spawn is set to True and the authorization_url is required.

mishaschwartz commented 5 months ago

@tlvu https://github.com/Ouranosinc/jupyterhub/pull/30 should cover this edge case

tlvu commented 5 months ago

Is JUPYTERHUB_CRYPT_KEY set? If so, then the MagpieAuthenticator.refresh_pre_spawn is set to True and the authorization_url is required.

@mishaschwartz Oh indeed I also had JUPYTERHUB_CRYPT_KEY set. I was trying out all the new options, didn't know they were mutually exclusive.

Confirmed that removing JUPYTERHUB_CRYPT_KEY while having JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="", I was able to login and spawn the server fine.

tlvu commented 5 months ago

Ouranosinc/jupyterhub#30 should cover this edge case

So I've read the PR, so the problem is not due to both JUPYTERHUB_CRYPT_KEY and JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL="" are mutually exclusive but due to a previous auth_state left behind?

So both of these options can be set at the same time and it should work?

mishaschwartz commented 5 months ago

So both of these options can be set at the same time and it should work?

Yeah exactly

tlvu commented 5 months ago

So both of these options can be set at the same time and it should work?

Yeah exactly

Just to be clear. Your fix is to make these 2 options not mutually exclusive. Otherwise, in the current state, they are mutually exclusive? Just so I fully understand the problem.

mishaschwartz commented 5 months ago

@tlvu

As of https://github.com/Ouranosinc/jupyterhub/pull/30 the behaviour is as follows:

JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL is set:
- only users that have permission to access jupyterhub (as defined by magpie service permissions) can log in to jupyterhub
- if JUPYTERHUB_CRYPT_KEY is set, jupyterhub will also:
  - check if the user still has permission when they spawn a server
  - check if the user still has permission at most every JUPYTERHUB_AUTHENTICATOR_REFRESH_AGE seconds
  - store the login cookies for the user in the jupyterhub database. A user can access these in their jupyterlab session to make accessing other resources that are protected behind twitcher easier.
JUPYTERHUB_AUTHENTICATOR_AUTHORIZATION_URL is not set:
- any user that exists in magpie is allowed to log in to jupyterhub
- JUPYTERHUB_CRYPT_KEY can be set or not and will have no effect

bird-house / birdhouse-deploy