terraform-aws-modules / terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources πŸ‡ΊπŸ‡¦
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
Apache License 2.0
4.47k stars 4.08k forks source link

The module ignores desired_size on change #2151

Closed rooty0 closed 2 years ago

rooty0 commented 2 years ago

Description

When I change desired_size in eks_managed_node_groups nothing happens. It looks like it works when a new resource is created; this issue is related to when a resource is created, and you're trying to change the value.

Versions

Terraform v1.2.3 on darwin_arm64 provider registry.terraform.io/hashicorp/aws v4.19.0 provider registry.terraform.io/hashicorp/cloudinit v2.2.0 provider registry.terraform.io/hashicorp/kubernetes v2.11.0 provider registry.terraform.io/hashicorp/random v3.3.1 provider registry.terraform.io/hashicorp/tls v3.4.0

EKS module version 18.26.3 (broken in 18.24.1 as well)

Reproduction Code [Required]

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

  # exec method is what is recommended by the Kubernetes provider
  exec {
    api_version = "client.authentication.k8s.io/v1alpha1"
    command     = "aws"
    # This requires the awscli to be installed locally where Terraform is executed
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_id]
  }
}

module "eks" {
  source       = "terraform-aws-modules/eks/aws"
  version      = "18.26.3"
  tags         = var.tags
  cluster_name = "core"

  create_cloudwatch_log_group = false

  eks_managed_node_group_defaults = {
    iam_role_use_name_prefix = false
  }

  cluster_addons = {
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    kube-proxy = {}
    vpc-cni = {
      resolve_conflicts = "OVERWRITE"
      addon_version     = "v1.11.2-eksbuild.1"
    }
  }

  node_security_group_additional_rules = {
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["0.0.0.0/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
    ingress_webhook_tcp = {
      description                   = "Control plane invoke ingress webhook"
      protocol                      = "tcp"
      from_port                     = 8443
      to_port                       = 8443
      type                          = "ingress"
      source_cluster_security_group = true
    }
    all_node_groups = {
      description = "Allow all traffic between worker node groups"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
      #      source_security_group_id = "sg-050be00d36699e32c"
    }
  }

  eks_managed_node_groups = {
    asg2 = {
      min_size       = 0
      max_size       = 3
      desired_size   = 0
      instance_types = ["t3.xlarge"]
      key_name       = "mykey"
    }
  }
  manage_aws_auth_configmap = true
  aws_auth_roles = [
    {
      rolearn  = "omitted"
      username = "aaa-admin"
      groups   = ["system:masters"]
    }
  ]

  vpc_id     = module.vpc.vpc_id
  subnet_ids = [module.vpc.public_subnets[0], module.vpc.public_subnets[1]]
}

Steps to reproduce the behavior:

I'm using my workspace us-west-2 I run terraform plan -out terraform.tfplan

Expected behavior

I'm expecting to see a change for desired_size, something like

      ~ scaling_config {
          ~ desired_size     = 0 -> 1
            # (2 unchanged attributes hidden)
        }

Actual behavior

getting No changes. Your infrastructure matches the configuration.

bryantbiggs commented 2 years ago

This module has made the stance of ignoring the desired changes because of the predominant use of cluster autoscalers (cluster-autoscaler, karpenter, etc.). Therefore, changing the desired size after initial deployment has to be made outside of the module

rooty0 commented 2 years ago

thank you @bryantbiggs , can't believe I miss it in the documentation

bryantbiggs commented 2 years ago

no worries!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

bryantbiggs commented 1 year ago

https://github.com/bryantbiggs/eks-desired-size-hack