Wierdness in ordering of pre_spawn_hook and load_user_options

[rkdarst]

I was just upgrading JH/kubespawner, and noticed that PR #301 caused a problem and possibly there's a bug in how it interacts with JupyterHub. #301 handles loading the user options the correct way (in a load_user_options function instead of options_from_form). Thanks for #301, by the way, that allows us to do better CI.

JH runs pre_spawn_hook first, then .start(), but .start() applies the profile options. So with the latest kubespawner the hook can't use the kubespawner_override settings (and would have to handle the raw user_options first). And kubespawner_override will also override anything that is also set in pre_spawn_hook.

I have worked around it by adding this to the top of my pre_spawn_hook, which applies the profile options, then eliminates the profile list. An false-like but not None value for profile list makes kubespawner skip applying the profile list again.

async def pre_spawn_hook(spawner):
    await spawner.load_user_options()
    spawner._profile_list = [ ]

I'm not sure what the "right" answer is, since this seems almost intrinsic to how JH is set up with options being applied in start and hook being done first. My first idea is in the default pre_spawn_hook which needs to be super()ed, but I don't know.

[consideRatio]

@rkdarst could you provide an update on this issue? Is this relevant still etc, and do you have a concrete action plan for this issue at this point?

[benjimin]

I think documentation for all the hooks should be consolidated into a section that better explains the sequencing, and that the API should also consider providing a hook that runs immediately before the pod is created.

For anyone wanting both user-specific customisations and profile choices (for example, if they want to inject the username into an environment variable, but also want the profile to set another environment variable) then they probably want the user-customisation to occur after the profile choice has been applied but before the pod manifest is produced. Any earlier and the profile clobbers the customisation; any later and manipulating spawner attributes has no effect. The problem is that there is actually no such hook.

Probably the best approach (currently) is to use modify_pod_hook but discard the pod object and instead return spawner.get_pod_manifest() (after applying final customisations to the spawner attributes).

(This seems simpler than the alternative approach of using pre_spawn_hook to load_user_options() early, then apply further user customisations, and then trying to hack the spawner so that reapplying load_user_options will take no effect, yet without breaking how the spawner will behave if the user tries to launch another server, with a different selection of choices, immediately after stopping this one.)

Otherwise, for anyone wanting to use the hooks to make customisations, they will probably need to know:

• Some surrounding context such as that an interactive Jupyterhub session (with authentication state) is represented by a User instance, which contains spawner instances for each each single-user server instance that is currently running or being configured for that individual account. And that the spawner instances serialise and deserialise through a persistent database (so that the hub can be killed and restarted without disrupting active user sessions).
• That the spawn sequence involves:
  1. post_auth_hook having had the opportunity to insert extra data into auth_state (which is a simple dict) after user authentication but before instantiating spawners;
  2. the spawner gets created, or alternatively the user's previous spawner instance (from a server that is no longer running) may get recycled;
  3. the authenticator may populate auth-related state into the spawner;
  4. auth_state_hook(spawner, auth_state:=User.get_auth_state()) facilitates authentication-related customisation;
  5. options_form is run (normally returning spawner._render_options_form(profile_list)) and the output is rendered and served as a web form;
  6. pre_spawn_hook is run;
  7. spawner.load_user_options() overrides attributes according to choices made by the user (applying a profile and its options);
  8. pod = modify_pod_hook(spawner, pod:=spawner.get_pod_manifest()) gets to manipulate the manifest, which is a complicated object composed of many kubernetes_asyncio classes;
  9. the pod manifest is submitted to the kubernetes API server, and if nothing has failed then after_pod_created_hook(spawner, pod) gets run;
• that post_stop_hook(spawner) runs after the pod will have been deleted;
• that lifecycle_hooks is a kubernetes field with no relation to spawner lifecycle;
• and that Jupyterhub also has other hooks separate from the spawners, like subdomain_hook which defines how usernames map to URLs.