Open sgibson91 opened 1 year ago
I read through https://github.com/jupyterhub/mybinder.org-deploy/blob/master/terraform/ovh/main.tf, and based on that here is the list of AWS resources that I think need to be created:
There are two ways to do this:
Both are valid ways to proceed, depending on the amount of complexity we want to sustain. In general, I personally prefer option (2), as I think it is clearer and has fewer levels of complex abstractions. It also matches how we have done it on GKE right now (https://github.com/jupyterhub/mybinder.org-deploy/blob/master/terraform/modules/mybinder/resource.tf).
However, I think in this specific case, (1) is also a workable alternative! This is primarily because the AWS setup for Kubernetes is far more fiddly than what exists for GKE, and that sucks. A lot of copypasta and 'gotcha's need to be managed. This is the precise place where terraform modules shine, so perhaps using those here is a good idea. https://github.com/hashicorp/learn-terraform-provision-eks-cluster has a fairly good, bare bones setup that does 1-4 in the list above easily. Maybe we can start there and go from there? The one modification I'd make to the example is that it has public and private subnets, and hence NAT enabled. We can survive with just a public subnet, and hence don't need NAT.
Thanks @yuvipanda! I think I had a hybrid idea between (1) and (2) which was to use the official aws provider for as much as possible, and then the module for the vpc network, because I think networking is the trickiest part. But perhaps that's unnecessary.
I think there's something missing from this list as well. Whatever the AWS "speak" is for something that will allow the cluster to push and pull images from the ECR. A role?
An IAM user that can be used by CI/CD to authenticate to the cluster for automated deployment
Can you use an IAM role and GitHub OIDC instead? This avoids needing hard-coded secrets, you'll get a temporary token every time. https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services It's been working well for me when deploying infrastructure on AWS.
Also https://github.com/jupyterhub/binderhub/pull/1055 (Add AWS ECR support (round two)) is probably relevant, I was going to review it but never had time š. It includes IAM instructions.
Can you use an IAM role and GitHub OIDC instead?
Let's keep this as a stretch goal. I'm learning and so want to focus on the MVP for now. This feels like something that can be switched to later on after we've got the basic infrastructure setup first.
Also jupyterhub/binderhub#1055 (Add AWS ECR support (round two)) is probably relevant, I was going to review it but never had time š. It includes IAM instructions.
Thank you
@sgibson91 at least in this case, I think the EKS setup is probably more complicated than the network stuff. But totally ok to start from using modules for VPC and go from there.
I've previously deployed a dev EKS cluster with https://github.com/hashicorp/learn-terraform-provision-eks-cluster It took a bit of digging into the EKS module code when I wanted to customise it though.
I opened this PR with my work so far
As discussed in https://github.com/jupyterhub/team-compass/issues/501, Pangeo want to simplify their Binder infrastructure by removing dask-gateway. This effectively makes it an "ordinary" binder and, therefore, a great candidate as a federation member. Especially since this will be deployed to AWS, which we don't currently cover. Since 2i2c are working with Pangeo (via a grant held at Columbia) to operationalise their cloud infrastructure, I will be taking the lead on the technical aspect of deployment, and this issue will be tracking that.
Things to do
terraform/gcp
folder, and then create aterraform/aws
folder for the new config