search

terraform-aws-modules / terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
Apache License 2.0
4.47k stars 4.08k forks source link

update of eks_managed_node_groups values result in always growing increase in MAX and desired capacity for ASGs #2047

Closed dmitry-mightydevops closed 2 years ago

dmitry-mightydevops commented 2 years ago

Description

Modification of the values eks_managed_node_groups results in constantly growing number of MAX capacity and desired capacity in ASG.

node group is created with 

min_size     = 1
max_size     = 5
desired_size = 1

Initial creation - all is good

The change

Modify block_device_mappings.xvda.ebs.volume_size 50 -> 60 and perform terraform apply results in iterative updates to the ASG for the nodegroup:

The plan

# module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["apps-v3"].aws_eks_node_group.this[0] will be updated in-place
  ~ resource "aws_eks_node_group" "this" {
        id              = "xxx-prod-eks:prod-apps-v3"
        tags            = {
            "Name"                                = "prod-apps-v3"
            "client"                              = "xxx"
            "cluster"                             = "xxx-prod-eks"
            "created"                             = "4/18/2022"
            "created_by"                          = "Michael"
            "environment"                         = "prod"
            "karpenter.sh/discovery"              = "xxx-prod-eks"
            "kubernetes.io/cluster/xxx-prod-eks" = "owned"
            "ops_team"                            = "Michael/Michaela"
            "ops_team_email"                      = "devops+xxx@example.com"
            "team"                                = "example"
            "terraform"                           = "true"
            "terraform_source"                    = "https://github.com/example/xxx-infra-aws"
            "updated"                             = "4/18/2022"
            "updated_by"                          = "Michael"
        }
        # (15 unchanged attributes hidden)

      ~ launch_template {
            id      = "lt-0053d426f85699916"
            name    = "apps-v3-20220429015545647000000017"
          ~ version = "1" -> (known after apply)
        }

        # (3 unchanged blocks hidden)
    }

  # module.eks_cluster.module.eks_cluster.module.eks_managed_node_group["apps-v3"].aws_launch_template.this[0] will be updated in-place
  ~ resource "aws_launch_template" "this" {
      ~ default_version         = 1 -> (known after apply)
        id                      = "lt-0053d426f85699916"
      ~ latest_version          = 1 -> (known after apply)
        name                    = "apps-v3-20220429015545647000000017"
        tags                    = {
            "client"                              = "xxx"
            "cluster"                             = "xxx-prod-eks"
            "created"                             = "4/18/2022"
            "created_by"                          = "Michael"
            "environment"                         = "prod"
            "karpenter.sh/discovery"              = "xxx-prod-eks"
            "kubernetes.io/cluster/xxx-prod-eks" = "owned"
            "ops_team"                            = "Michael/Michaela"
            "ops_team_email"                      = "devops+xxx@example.com"
            "team"                                = "example"
            "terraform"                           = "true"
            "terraform_source"                    = "https://github.com/example/xxx-infra-aws"
            "updated"                             = "4/18/2022"
            "updated_by"                          = "Michael"
        }
        # (9 unchanged attributes hidden)

      ~ block_device_mappings {
            # (1 unchanged attribute hidden)

          ~ ebs {
              ~ volume_size           = 50 -> 60
                # (6 unchanged attributes hidden)
            }
        }

        # (5 unchanged blocks hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

ASG iteratively being updated with the following values:

max increased to 6 image max increased to 7 image max increased to 8 image max increased to 9 image

instances: image

Then it starts cooling down

It this a normal and expected behavior? If I change the "key name" in the eks_managed_node_groups - then the old nodegroup is deleted and new one is created. And overall the procedure is much faster.

I also got the same issue when my tags are updated. So if I change just tags, let's say

"created"  = "4/18/2022" to "created"  = "4/19/2022"

then I get a lot of EC2 instances spawned and it is very time consuming operation if max_size > 1

Overall the "apply" took 20minutes.

Versions

- Provider version(s):

## Reproduction Code [Required]

I create the eks cluster with the following config:

```hcl

module "eks_ssh_sg_label" {
  source     = "git::https://github.com/cloudposse/terraform-null-label.git?ref=0.25.0"
  namespace  = var.cluster_name
  attributes = ["ssh", "access", "sg"]
}

data "aws_eks_cluster" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks_cluster.cluster_id
}

data "aws_availability_zones" "available" {}
data "aws_caller_identity" "current" {}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

resource "aws_kms_key" "eks" {
  description             = "Customer managed key to encrypt kubernetes secrets in the cluster"
  deletion_window_in_days = 7
  enable_key_rotation     = true

  tags = var.tags
}

resource "aws_kms_alias" "eks" {
  name          = "alias/${var.cluster_name}-kms-key"
  target_key_id = aws_kms_key.eks.key_id
}

resource "aws_security_group" "ssh" {
  count = var.allow_ssh_access ? 1 : 0

  name   = module.eks_ssh_sg_label.id
  vpc_id = var.vpc_id
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = var.allow_ssh_access_cidr
  }

  tags = var.tags
}

module "eks_cluster" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.20.5"

  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version
  vpc_id          = var.vpc_id
  subnet_ids      = var.subnet_ids

  cluster_endpoint_private_access = var.cluster_endpoint_private_access
  cluster_endpoint_public_access  = var.cluster_endpoint_public_access
  # cluster_endpoint_private_access_cidrs =
  cluster_endpoint_public_access_cidrs = var.cluster_endpoint_public_access_cidrs

  cluster_addons = {
    coredns = {
      resolve_conflicts = "OVERWRITE"
    }
    kube-proxy = {}
    vpc-cni = {
      resolve_conflicts = "OVERWRITE"
    }
  }

  cluster_encryption_config = [{
    provider_key_arn = aws_kms_key.eks.arn
    resources        = ["secrets"]
  }]

  node_security_group_additional_rules = {
    ingress_karpenter_webhook_tcp = {
      description                   = "Control plane invoke Karpenter webhook"
      protocol                      = "tcp"
      from_port                     = 8443
      to_port                       = 8443
      type                          = "ingress"
      source_cluster_security_group = true
    }

    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }
    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = ["0.0.0.0/0"]
      ipv6_cidr_blocks = ["::/0"]
    }
  }

  manage_aws_auth_configmap = true
  aws_auth_accounts = var.grant_access_iam_accounts
  aws_auth_roles    =  local.aws_auth_roles
  aws_auth_users    = local.aws_auth_users

  eks_managed_node_group_defaults = {

    disk_size = 50
    vpc_security_group_ids = var.allow_ssh_access ? [aws_security_group.ssh[0].id] : []

    post_bootstrap_user_data = <<-EOT
      cd /tmp
      sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
      sudo systemctl enable amazon-ssm-agent
      sudo systemctl start amazon-ssm-agent
      EOT

    iam_role_additional_policies = flatten([
      "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
      var.iam_role_additional_policies,
    ])

    use_name_prefix = false
  }

  self_managed_node_groups = {}
  eks_managed_node_groups = {

    // ▩ APPS ###############################################################################################
    apps-v3 = {
      name         = "prod-apps-v3"
      min_size     = 1
      max_size     = 5
      desired_size = 1
      subnet_ids = [
        "subnet-0235597d55d03df7f",
        "subnet-09f9ef6e5d66fd4ac",
        "subnet-026382e532aaed88c"
      ]
      instance_types = ["t3.medium"]

      labels = {
        apps     = "true"
        workload = "apps"
      }
      taints = []

      ebs_optimized = true
      capacity_type = "ON_DEMAND"

      iam_role_additional_policies = [
        "arn:aws:iam::xxx:policy/xxx-prod-eks-ecr-policy",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-s3-policy",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-secretmanager-policy",
        "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
        "arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-efs-policy",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-backup-policy"
      ]

      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 50
            volume_type           = "gp3"
            encrypted             = true
            kms_key_id            = "arn:aws:kms:us-east-1:xxx:key/f2bcff18-395c-47c6-a442-7f859285b6e0",
            delete_on_termination = true
          }
        }
      }

      create_security_group = false
    }

    // ▩ APPS SPOT / KARPENTER ##############################################################################
    karpenter-v3 = {
      name         = "prod-karpenter-v3"
      min_size     = 1
      max_size     = 5
      desired_size = 1
      subnet_ids = [
        "subnet-0235597d55d03df7f",
        "subnet-09f9ef6e5d66fd4ac",
        "subnet-026382e532aaed88c"
      ]
      instance_types = ["t3.medium"]

      labels = {
        apps      = "true"
        workload  = "apps"
        spot      = "true"
        karpenter = "true"
      }
      taints = []

      ebs_optimized = true
      capacity_type = "SPOT"

      iam_role_additional_policies = [
        "arn:aws:iam::xxx:policy/xxx-prod-eks-ecr-policy",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-s3-policy",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-secretmanager-policy",
        "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
        "arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-efs-policy",
        "arn:aws:iam::xxx:policy/xxx-prod-eks-backup-policy"
      ]

      block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = 50
            volume_type           = "gp3"
            encrypted             = true
            kms_key_id            = "arn:aws:kms:us-east-1:xxx:key/f2bcff18-395c-47c6-a442-7f859285b6e0",
            delete_on_termination = true
          }
        }
      }

      create_security_group = false
    }
  }

  enable_irsa = var.enable_irsa

  tags = merge(var.tags, {
    cluster                                     = var.cluster_name
    "karpenter.sh/discovery"                    = var.cluster_name
    "kubernetes.io/cluster/${var.cluster_name}" = "owned"
  })
}

this creates:

EC2 image

ASG image

Launch Templates image

Expected behavior

A new and single EC2 instance to be added if current number of instances is 1.

Actual behavior

Many EC2 instances added incrementally by updating the ASG, which is very time consuming, especially if max_size is >1

kaykhancheckpoint commented 2 years ago

I think i have experienced something similar :eyes:

bryantbiggs commented 2 years ago

@dmitry-mightydevops this is most likely due to the rolling update config for EKS managed node groups. Can you try setting max_unavailable or max_unavailable_percentage to 0 https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group#update_config-configuration-block

tanvp112 commented 2 years ago

The valid range for max_unavailable is 1 - 100. A cluster with desired instance of 1, min 0 and max 3 will hike to desired instance of 6 and max 8 instances! Then it eventually (~30mins) shrink back to desired instance of 1 and max 3 - all for the sake of updating 1 node in this case.

Any idea how to limit the number of excessive instance?

bryantbiggs commented 2 years ago

Closing since this is not a module issue but a configuration setting. Please see https://docs.aws.amazon.com/eks/latest/userguide/managed-node-update-behavior.html

dmitry-mightydevops commented 2 years ago

So I tested and max_unavailable and it has no effect - still getting a lot of nodes added and then removed. Changing new group (update ops) becomes very time consuming =~20-30m and it's much faster to just add a new entry in the eks_managed_node_groups, then kordon/drain and then remove the original nodegroup.

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.