Deploy and operate a BinderHub for Pangeo

[choldgraf]

Description / problem to solve

Problem description The Pangeo BinderHub has been down for about a month (due to crypto mining, but also because it did not have operational support to keep it going sustainably). The Pangeo community made heavy of use their Binder deployment, and it powered a lot of reproducible sharing (e.g., via gallery.pangeo.io.).

Proposed solution We should deploy a BinderHub on the 2i2c deployment infrastructure that can live in parallel to the JupyterHub we run for the Pangeo community. We'll need to make a few modifications to their setup (including using up-to-date binderhub versions and locking down auth more reliably).

What's the value and who would benefit This would allow the Pangeo community to re-gain the use of their BinderHub, which would benefit many people!

Implementation guide and constraints

There are a few things that we should consider here:

• We'll need to update our configuration and CI/CD infrastructure to be able to deploy a BinderHub chart, since we're currently assuming JupyterHub.
• We want to make sure we don't deploy a BinderHub and immediately run into the same crypto issues that Pangeo ran into before

Here's a GitHub issue where @scottyhq describes the environment that was available on the Pangeo BinderHub: https://github.com/pangeo-data/pangeo-binder/issues/195#issuecomment-989107771

Updates and ongoing work

Here are a few major issues that would need to be tackled as part of this effort:

• [x] https://github.com/2i2c-org/infrastructure/issues/857 (so that we can generalize our config beyond just JupyterHub)
• [x] https://github.com/2i2c-org/infrastructure/issues/879 (if we want to strengthen the CI/CD infrastructure in general before deploying new things, we might decide to reduce the scope a bit if need be)
• [x] https://github.com/2i2c-org/infrastructure/issues/1280
• [ ] https://github.com/2i2c-org/meta/issues/279
• [x] https://github.com/2i2c-org/infrastructure/issues/328
• [x] #1823
• [x] #1824

Admin

• Champion: @sgibson91
• Supporter/Collaborator: @octocat
• Project board: https://github.com/orgs/2i2c-org/projects/31

[sgibson91]

Just an update to this thread. The credits Scott offered have now gone https://discourse.pangeo.io/t/aws-pangeo-jupyterhubs-to-shut-down-friday-march-17/3228 So we need to figure out how else to fund a Pangeo Binder.

[choldgraf]

I think that means that the funding would need to come from the Columbia grants themselves, is that right? (maybe @rabernat can comment?)

If that is the case then I think we have two options[^1].

1. Deploy on a Columbia cloud project
   • Unclear to me if this is an option. Maybe @2i2c-org/engineering thinks of it as a non-starter? Please advise.
2. Deploy on a Pangeo cloud project.
   • Depends on https://github.com/2i2c-org/meta/issues/279

I likely don't have the capacity to spearhead this, so we'll need somebody (@jmunroe @colliand @damianavila) to track and move this forward.

[^1]: Assuming there's not some other pot of money to fund the infrastructure.

[pnasrat]

As I understand from onboarding it not all engineers have columbia accounts, and we don't have a clear process to request. From my perspective if the whole team is not able to support a cluster/hub then it is not sustainable to have just limited access to it.

Ref 2i2c-org/infrastructure#1799

[sgibson91]

Also, it appears Yuvi and I can no longer login to the Columbia emails we have anyway. Yuvi sent an email to Ryan and Julius. So major problems going to Columbia account route.

[damianavila]

Yep, setting up in Columbia land is a no-go. This meta issue will deal with the pieces needed for a pass-through.