[Initiative] Ephemeral sessions

yuvipanda commented 5 months ago

(This was manually migrated from https://github.com/2i2c-org/meta/issues/1105, since that was private and we want this to be public)

This was refined by me, @Gman0909 and @jmunroe. ProductBoard link.

Description

Provide the ability to spin up MyBinder-like ephemeral hubs to allow authenticated and unauthenticated users (based on configuration) access to the contents of an externally published artifact (GitHub repo, Zenodo, etc) animated via an interactive computing session.

User personas

2i2c engineering: sets up the ephemeral session functionality for a community. Admin: member of the community, sets up permissions globally, such as whether users are allowed to share to unauthenticated individuals, and whether to limit sharing to GitHub repos controlled by their organization. They will also set resource requests and limitations. Content creator/Sharer: has to be able to package a snapshot of their work to be shared as a link, including all necessary environment configurations to ensure the work can be reproduced and experienced in its full form by the recipient of the link. If the Admin has made unauthenticated access to shared links available, the sharer should have that option as a toggle when creating that link. Reader: receives a link, clicks it, and is brought into an interactive session that reproduces the work shared by the Content Creator/Sharer

Desired features

2i2c engineering

Everything we build should be upstreamable as much as possible

Admin

Can determine users and user groups who can have access to the infrastructure
- Includes ability to provide fully unauthenticated users
Can determine what kinda content can be run on the ephemeral hubs
- Restricted by binderhub provider specs - (eg. GitHub orgs / list of repos, zenodo DOIs, etc)
Basic usage monitoring dashboard
Can decide to deploy or not deploy dask-gateway

Implementation note: All admin actions will have to currently go through 2i2c support.

Creator

Ability to create a link for sharing their work
Specify the environment their work needs
Specify the content of their work, if it's separate from where the environment is hosted
Ability to select what kinda resources the Reader receives when they click the link

Reader

Receives a link, clicks it, and is brought into an interactive session that reproduces the work shared by the Content Creator/Sharer

Why is it important

Compliant with our Value Proposition's support of creating and sharing knowledge, we want to empower our communities to more easily permit access to their works, in the form of interactive computing sessions, to authorized and unauthorized users, with zero-setup on the part of the recipient of that shared work. This is also a commonly requested feature we have committed to delivering to the likes of NASA HPOSS and Project Pythia

Measures and Definition of Done

Communities are able to easily share work to authorized and unauthorized users via a link leading to a zero-setup interactive compute session containing said work.

## Tasks
- [ ] https://github.com/2i2c-org/infrastructure/issues/4085
- [ ] https://github.com/2i2c-org/infrastructure/issues/4086
- [ ] https://github.com/2i2c-org/infrastructure/issues/4133
- [ ] https://github.com/2i2c-org/infrastructure/issues/4168
- [ ] https://github.com/2i2c-org/infrastructure/issues/4197
- [ ] https://github.com/2i2c-org/infrastructure/issues/4169
- [ ] https://github.com/2i2c-org/infrastructure/issues/4167
- [ ] https://github.com/2i2c-org/infrastructure/issues/4195
- [ ] https://github.com/2i2c-org/infrastructure/issues/4134
- [ ] https://github.com/2i2c-org/binderhub-service/issues/95
- [ ] https://github.com/yuvipanda/jupyterhub-fancy-profiles/issues/37
- [ ] https://github.com/2i2c-org/infrastructure/issues/4242
- [ ] https://github.com/2i2c-org/infrastructure/issues/4113
- [ ] https://github.com/2i2c-org/infrastructure/issues/4087
- [ ] https://github.com/2i2c-org/binderhub-service/issues/57
- [ ] https://github.com/2i2c-org/infrastructure/issues/4246
- [ ] https://github.com/2i2c-org/infrastructure/issues/4247
- [ ] https://github.com/2i2c-org/infrastructure/issues/4301
- [ ] https://github.com/2i2c-org/infrastructure/issues/4309
- [ ] https://github.com/2i2c-org/infrastructure/issues/4310
- [ ] https://github.com/2i2c-org/infrastructure/issues/4365
- [ ] https://github.com/2i2c-org/infrastructure/issues/4370
- [ ] https://github.com/2i2c-org/infrastructure/issues/4371
- [ ] https://github.com/2i2c-org/infrastructure/issues/4241
- [ ] https://github.com/2i2c-org/infrastructure/issues/4423

yuvipanda commented 5 months ago

Porting some relevant comments from the internal issue

From @yuvipanda, at https://github.com/2i2c-org/meta/issues/1105#issuecomment-2111573907

I had a nice and productive scoping session with @consideRatio today. I'll create an epic in the infrastructure repository later, but wanted to dump the first set of tasks we have scoped out so far:

Remove all of binderhub chart
Spike to figure out what is needed in binderhub-service to expose classical binderhub UI including ability to launch in a way that makes it composable. Constraint is that binderhub-service should not know what is jupyterhub, but can have extensibility that is generic added to it. Outcome should primarily be: documentation changes, and possibly extra extensibility in binderhub-service.
Explicitly specify constraints for use of fancy-profiles vs binderhub UI. Persistent home directory is an important one here.
Explicitly specify scope for binderhub-service.
Spike for figuring out how dask-gateway can be an optional add on in basehub rather than daskhub.

These all need to be cleaned up and expanded, a task for tomorrow.

From @yuvipanda in https://github.com/2i2c-org/meta/issues/1105#issuecomment-2113090855

I've updated the issue body with the tasks from https://github.com/2i2c-org/meta/issues/1105#issuecomment-2111573907. They're also prioritized in the order I believe they should be done. This should allow us to make progress on a lot of these goals! While I don't believe we'll have a full deployment once these are done, it would allow us to spec out a 'new hub' issue that would allow us to get there by the end of the sprint I believe.

The one thing that's currently explicitly out of scope is 'resource selection' and 'GPU' use, which are tied together. This would involve reviving these upstream conversations:

From @Gman0909 in https://github.com/2i2c-org/meta/issues/1105#issuecomment-2114375304

Thanks Yuvi. Given the likely time lag that is involved in any upstream involvement, should we not prioritize getting some of these conversations going?

Also, if we expect that the current task list will lead to the speccing out of a new issue that will allow us to meet the DoD, I feel we should record that in the task list, even as a draft if need be.

yuvipanda commented 5 months ago

Me and @jmunroe have refined #4133 to be the first demo hub to be deployed with this functionality, and we believe it's fully refined now. Our goal was to make sure that if we can deploy this, we can deploy what we promised project pythia. I've prioritized it above our dask-gateway work.

yuvipanda commented 5 months ago

I have spoken with @jmunroe and @Gman0909, and we're going to split off the following tasks into their own initiative:

https://github.com/2i2c-org/infrastructure/issues/4151 (this is not currently required by the three grants that drive this inititative)
https://github.com/2i2c-org/infrastructure/issues/4152 (this shall be moved to be a part of completing our GESIS work as well as upcoming NASA VEDA work)

yuvipanda commented 5 months ago

https://github.com/2i2c-org/infrastructure/issues/4246 and https://github.com/2i2c-org/infrastructure/issues/4247 need further refinement, and I'd like them to be done by someone who wasn't the original set of people working on this.

Gman0909 commented 5 months ago

Thanks @yuvipanda. I'm assuming that, funding-centered priorities notwithstanding, the plan is to get these two done contiguously to the rest of the work on Ephemeral Hubs?

GeorgianaElena commented 5 months ago

and I'd like them to be done by someone who wasn't the original set of people working on this.

@yuvipanda, the second iterartion of https://infrastructure.2i2c.org/howto/features/binderhub-ui/ is now ready so this should unblock other people doing those deployments

yuvipanda commented 5 months ago

@Gman0909 yes. I've spoken to @sgibson91 and she'll pick those up (after a session as part of https://github.com/2i2c-org/infrastructure/issues/4113)

yuvipanda commented 4 months ago

https://github.com/2i2c-org/infrastructure/issues/3655 is a big milestone, as it's now counted as 'tech support' than product dev. Upon completion of that without issues, I'd say we can then proceed to add this to our single source of truth about the kind of hubs we can deploy! I'll work with @Gman0909 on that next week.

yuvipanda commented 4 months ago

I don't think there are any more tasks to do for this initiative - once the two remaining ones are completed, we are good to go! We need to add this to the canonical list of things we can provide, and then mark it as done! \o/

yuvipanda commented 4 months ago

All tasks are complete here.

I've added BinderHub UI to https://docs.google.com/spreadsheets/d/1qy_ozom5QBSiDDedzmxHUsBWlEot44gl3qPWfvMYiEA/edit?gid=0#gid=0 which IIRC is our 'canonical' location for various components we 'support'.

With that, I decree this initiative completed!

Congratulations to all of us :)

2i2c-org / infrastructure