Refine work on upgrading to z2jh 4 (JupyterHub 5, KubeSpawner 7, OAuthenticator 17)

[consideRatio]

Z2JH 4.0.0 is now out (changelog). This issue tracks refining work to make it get done.

I've already trialed upgrading z2jh from 3 to 4 (beta-4) for a 2i2c deployment (https://github.com/2i2c-org/infrastructure/issues/4916, https://github.com/2i2c-org/infrastructure/pull/4979), and I figure its reasonable for 2i2c to upgrade all hubs to z2jh 4.0.0 at this point in time.

---

Misc pointers

• Once a hub is upgraded, the jupyterhub database file can't be used again with an older version, so if you upgrade and need to roll back to z2jh 3 with JupyterHub 4, then you need to manually switch back to a JupyterHub .sqlite database file used before upgrading to z2jh 4.

  Below is a comment from https://github.com/2i2c-org/infrastructure/pull/4979 how this can be done if needed:

  • Downgrading after upgrade couldn't be done without first replacing the jupyterhub.sqlite file with a backup. This can be manually handled by kubectl debug <hub pod name> -it --copy-to=test-debug --image=ubuntu -c hub -- /bin/bash and then looking at /srv/jupyterhub for sqlite backup files and using a backup instead of the one namedjupyterhub.sqlite`

• We lack a system for deploying a mix of z2jh chart versions, which makes us need to disable the CI system if we manually deploy hub by hub and test things while doing it. I think a GitHub based CI system can be disabled via the "..." button in https://github.com/2i2c-org/infrastructure/actions/workflows/deploy-hubs.yaml which could be relevant to do.
• Places were we specify subPath needs to be updated, instead of declaringsubPath: "{username}", we need it to be subPath: "{escaped_username}" to avoid introducing a new folder for each user in existing NFS storage.
• Watch out for authentication issues as we lack tests for it, make sure that you can sign in as before in various hubs after deploy.
• User servers running while the upgrade was made with JupyterHub 3, 4, and 5 in their images were accessible after jupyterhub was successfully restarted. This means that user images doesn't need to follow to JupyterHub 5 alongside this server side upgrade.

Definition of done

• Other issues have been created and refined to drive us towards upgrading to z2jh 4 more concretely.

[yuvipanda]

We would need https://github.com/yuvipanda/jupyterhub-fancy-profiles/pull/72 merged and deployed as well before we can upgrade