Objective JH-4: Group based shared folders

batpad commented 2 months ago

Motivation

Currently, we have the following 2 sets of shared folders available for all users:

shared , available read-only for all users. The exact same folder is mounted as shared-readwrite only for admins. Admins can write out things that can then be read by all users
shared-public, available as read-write for all users. Nothing prevents users from accidentally trampling on each others’ files here.

While GitHub repositories are the primary means for groups to collaborate on code, it’s still very useful to have a lightweight way for users within a group to have a space just for themselves. This was asked for by at least two separate groups (EIS Fire and the GHG workshop that was supported) during the last quarter.

Proposal

For groups that ask for it, we set up shared/ directories that are visible (read-write) only to members who belong to that group. This would be similar to the shared-public directories, but group restricted.

Owners

@yuvipanda (2i2c)
@batpad (Development Seed)

Success Criteria

[ ] Groups can request a shared directory be set up just for themselves
[ ] Only users who belong to this group can see the shared directory

yuvipanda commented 2 months ago

The way to realize this is to set up extra volumeMounts based on user's group membership. This would be doable with https://github.com/jupyterhub/jupyterhub/pull/4822. That was merged last month, and should be in a release soon.

However, volume_mounts (https://github.com/jupyterhub/kubespawner/blob/593405d24af017bc759b6763a0a2af8af4de0f38/kubespawner/spawner.py#L1066) is what needs to be overrriden, and it's a List not a Dict. This means we can not simply add more volume mounts based on user group membership - only replace the whole list. This is because Lists can not be merged, only dictionaries can be merged.

So we'd need to allow for volume_mounts to be either a List (for backwards compatibility) or a Dict.

So to complete this objective, we'll need to:

[x] Make a PR to kubespawner that converts all List traitlets in https://github.com/jupyterhub/kubespawner/blob/main/kubespawner/spawner.py to Union, with allowed types of List and Dict. If it's a dict, it should be sorted lexicographically (like in https://github.com/jupyterhub/jupyterhub/pull/4822/files#diff-cfa168c3c78c01c2bbdbd43bfbc61d7e2bf150b3f4932cea573283a2a9e1a684R1562) and the items be used as value. This allows for easy overriding, is backwards compatible, and reviewable easily! Should add a test as well.
[ ] Make a PR to https://github.com/jupyterhub/zero-to-jupyterhub-k8s that allows for these config to be set as Dicts as well. This should involve changes in https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/files/hub/jupyterhub_config.py (any custom handling), https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/values.yaml (default values changed from list to dict), and https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/main/jupyterhub/values.schema.yaml (schema adjusted to accept dicts and list). Documentation should also updated where relevant.
[ ] Get a release of jupyterhub out, of kubespawner out, of z2jh out
[ ] Update 2i2c infra to use these new releases
[ ] Setup config to create shared directories (similar to admin readwrite directories now)

This will take a bit of time, although the total amount of work done is limited.

One of my goals is to find more ways for @sunu (or @batpad?) to contribute to the Jupyter ecosystem, with the eventual goal of becoming a maintainer (if that is something you want, of course :)). To that end, I think it would be useful for you to do steps (1) and (2)? I can either review them myself, or find time for review from others.

batpad commented 1 week ago

@sunu @yuvipanda I think we hit our goals for this quarter in terms of making the required changes to upstream projects to be able to do this work. I believe we now need to wait for releases of the upstream projects to be able to configure and integrate onto the hubs. Could one of you give a quick update of where we are with that and what this needs to get over the line to be implemented on our hubs?

NASA-IMPACT / veda-jupyterhub