search

pangeo-data / terraform-deploy

deployment of pangeo jupyterhub infrastructure with terraform
MIT License
19 stars 31 forks source link

Multiple Clusters and Idempotency #21

Open super-cob opened 4 years ago

super-cob commented 4 years ago

when I attempt to run this after the cluster-autoscaler IAM role already exists, I get this error 

Error: Kubernetes cluster unreachable

  on autoscaler.tf line 63, in resource "helm_release" "cluster-autoscaler":
  63: resource "helm_release" "cluster-autoscaler" {

Error: Post https://4663658293E17DFB6A45DCF1D286FB08.gr7.us-east-1.eks.amazonaws.com/api/v1/namespaces: dial tcp 54.164.173.32:443: i/o timeout

  on efs.tf line 37, in resource "kubernetes_namespace" "support":
  37: resource "kubernetes_namespace" "support" {

Error: Error creating IAM Role cluster-autoscaler: EntityAlreadyExists: Role with name cluster-autoscaler already exists.
    status code: 409, request id: 853efb00-7b45-4b2d-951b-940d723ccd3b

  on .terraform/modules/iam_assumable_role_admin/terraform-aws-iam-2.6.0/modules/iam-assumable-role-with-oidc/main.tf line 43, in resource "aws_iam_role" "this":
  43: resource "aws_iam_role" "this" {

I was wondering if it would be useful and possible for this to be idempotent, so that it would detect the IAM role exists already and skip creating it. This would allow multiple clusters in one account, our current use case for this is that multiple people are trying to work through setting a cluster up to become familiar with it.

salvis2 commented 4 years ago

@super-cob I think you can achieve this as-written if you do two things:

  1. Make sure the cluster-autoscaler is already managed by Terraform.
  2. Enable a remote backend for Terraform. I have some configuration that sets up an S3 bucket to hold the current Terraform configuration and a DynamoDB table to enable locking.

Notes on each of those:

super-cob commented 4 years ago

Oh cool, can I use something you've written as an example? I'd like to try this

On Fri, Apr 3, 2020 at 4:00 PM Sebastian Alvis notifications@github.com wrote:

@super-cob https://github.com/super-cob I think you can achieve this as-written if you do two things:

  1. Make sure the cluster-autoscaler is already managed by Terraform.
  2. Enable a remote backend for Terraform. I have some configuration that sets up an S3 bucket to hold the current Terraform configuration and a DynamoDB table to enable locking.

Notes on each of those:

  • Terraform is really good at managing / interacting with things it has already setup and not very good at noticing what already exists if you made it another way, eg. AWS Console.
  • The remote backend is a really good thing to setup if you plan on having multiple manage the cluster anyways.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pangeo-data/terraform-deploy/issues/21#issuecomment-608631808, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOZ23BD4BL6D4AF43U242RTRKY56JANCNFSM4L4JTV3Q .

yuvipanda commented 4 years ago

I just worked with @super-cob a little on this. Based on my understanding of how backends work, the use case is slightly different - we want to have two different instances of the hub infrastructure in the same AWS account. Currently, we don't prefix IAM roles, etc with the cluster name. This means you can't have two instances of this terraform setup on the same AWS account... This is separate from multiple people managing the same hub, which is (IIRC) what the backends fix.

Similar to how in helm we prefix everything with the name of the release (or at least we should), I think we should prefix everything with the name of the cluster here.

yuvipanda commented 4 years ago

We could also use https://www.terraform.io/docs/providers/random/r/pet.html or https://www.terraform.io/docs/providers/random/r/id.html, but for our usage I prefer prefixing instead.

yuvipanda commented 4 years ago

See https://medium.com/@pavloosadchyi/terraform-patterns-and-tricks-i-use-every-day-117861531173 talking about prefixes. I think this matches our use case very well