Technical checklist for joining the BinderHub Federation

sgibson91 commented 5 years ago

As I'm working through adding the Turing to the Federation, I thought I'd create a checklist of the technical things that need to happen and maybe we can have a discussion on the minimum configuration needed too (e.g. what to do about grafana, matomo, etc.).

Please feel free to add to or expand the below!

Checklist

In a branch/fork of the mybinder.org-deploy repo:

[ ] Add a YAML file to config with your BinderHub configuration.
- Minimum requirements for this file:
- [ ] Your hub URL
- [ ] Your container registry URL
- [ ] Your container registry token URL for OAuth
- [ ] Ingress annotations for subdomains (Binder and JupyterHub, etc)
- [ ] Resource requests
- [ ] Extra Volumes
- [ ] JupyterHub Resources
- [ ] Single user limits
- [ ] Proxy
- [ ] Scheduling
- [ ] grafana ingress and data sources
- [ ] prometheus ingress
- [ ] nginx-ingress config
- Top level keys not required:
- proxyPatches
- federationRedirect
[ ] Add a YAML file to secrets/config with secret configuration
- Minimum requirements for this file:
- [ ] Container registry username
- [ ] Container registry password
- [ ] API token
- [ ] Secret token
[ ] Add your cluster's authentication procedure to deploy.py
- Minimum requirements for this file:
- [ ] A function containing commands to access your cluster
- [ ] An if/elif statement in main handling your cluster
- [ ] Add a file with your authentication info to secrets
[ ] Add an extra run of deploy.py to .travis.yml
- Minimum requirements for this file:
- [ ] If necessary, add an extra step to stage 1 installing any required CLIs
- [ ] Add an extra step to stage 5 executing deployment on your cluster
- [ ] Add an extra step to stage 5 verifying the deployment on your cluster

Questions

Is a config/<deployment>/<deployment>_mybinder_org_ingress.yaml file necessary? What's its purpose?
- Yes, this is how e.g. OVH redirects binder.mybinder.ovh to ovh.mybinder.org. If managing your own subdomain, this will be required.

Other important stuff

RE deploy.py args:

"release" is the name of the helm thing being deployed, and "cluster" is the name of the kubernetes cluster configuration. We have assumptions about what those two mean in a number of places, and it's definitely not clear everywhere. The only way to be really clear is to have no names re-used
Does the new cluster use kube-dns or coredns? secrets/ban.py may have trouble.

sgibson91 commented 5 years ago

Could we create a "federation template"? E.g. fill out these YAML files, the values that are already filled are what the BinderHub Federation expects for a consistent UX, blank fields are for your own keys/subdomains/other.

manics commented 5 years ago

There's some duplicated config between each binderhub.... is there enough in common to be worth having a common config which each binderhub overrides as necessary?

sgibson91 commented 5 years ago

The easiest way to achieve that would be to put the common config in mybinder/values.yaml. It'd be interesting to know the rest of the team's opinion on that.

manics commented 5 years ago

I was thinking of a common config in this repo and take advantage of Helm's ability to merge multiple configs (latest takes precedence)

yuvipanda commented 5 years ago

Could we create a "federation template"? E.g. fill out these YAML files, the values that are already filled are what the BinderHub Federation expects for a consistent UX, blank fields are for your own keys/subdomains/other.

At UC Berkeley, we use an internal cookiecutter for this. See http://docs.datahub.berkeley.edu/en/latest/howto/new-hub.html for docs, and https://github.com/berkeley-dsep-infra/datahub/tree/staging/deployments/template for our template.

sgibson91 commented 5 years ago

I was thinking of a common config in this repo and take advantage of Helm's ability to merge multiple configs (latest takes precedence)

Would that work given that the travis file updates a cluster at a time due to the different auth requirements?

Thanks @yuvipanda, that looks very cool!

manics commented 5 years ago

Would that work given that the travis file updates a cluster at a time due to the different auth requirements?

I image so... deploy.py already passes multiple configs to Helm: https://github.com/jupyterhub/mybinder.org-deploy/blob/cd7fb5b6839d0c17c42500eb7faec198337bdaac/deploy.py#L136-L138

sgibson91 commented 5 years ago

Ah, yes sorry. My brain wasn't present when I wrote that!

betatim commented 5 years ago

What do you mean with "template for the federation"?

Looking at https://github.com/jupyterhub/mybinder.org-deploy/blob/master/config/prod.yaml I think an individual cluster probably doesn't need the following top level keys: static, proxyPatches, redirector, matomo, gcsProxy or federationRedirect.

I'd definitely run a prometheus instance (so we can add it as a data source to grafana.mybinder.org). A local grafana is probably a good idea as well but can get away without one.

Off the top of my head I don't know if we need https://github.com/jupyterhub/mybinder.org-deploy/blob/master/config/ovh/ovh_mybinder_org_ingress.yaml or not. I thought listing the hostname inn the ingress already defined in ovh.yaml would be enough.

sgibson91 commented 5 years ago

What do you mean with "template for the federation"?

I mean a blank config file that can just be filled in to add a new cluster to the federation. It contains only the necessary keys required. Something like:

binderhub:
  config:
    hub_url: PUT_YOUR_JUPYTERHUB_URL_HERE

  registry:
    url: PUT_YOUR_REGISTRY_URL_HERE

etc...

I'm making a lot of assumptions around what's in the OVH deployment being "correct" and "necessary".

I'd definitely run a prometheus instance (so we can add it as a data source to grafana.mybinder.org). A local grafana is probably a good idea as well but can get away without one.

Is there a recommended guide on how to set this up?

betatim commented 5 years ago

You will get a basic grafana and prometheus via the mybinder chart as that chart depends on those two. Was more thinking out loud which of the top level keys should and shouldn't be copied/are relevant.

The best we have is looking at the OVH config and the gke config. There are differences which are part "gke" vs "ovh" and part "gke does stuff that other federation members don't do". Which is annoying but :-/

sgibson91 commented 5 years ago

Yeah, I'd be happy documenting those differences somewhere for the next federation member :)

sgibson91 commented 5 years ago

One action to take would be to refactor the deployment configs so we have one prod.yaml that is applied to every member and then have a gke.yaml, turing.yaml, ovh.yaml with their specific overrides.

(paraphrasing a gitter chat with @minrk)

meeseeksmachine commented 4 years ago

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/how-can-i-support-mybinder-org/4692/4

jupyterhub / team-compass