terraform-aws-modules / terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
Apache License 2.0
4.39k stars 4.04k forks source link

Launch template block_device_mapping not working #2699

Closed DarkoDevops closed 7 months ago

DarkoDevops commented 1 year ago

Description

I have EKS on version "18.29.0" and need to upgrade to the latest. I created launch template:

resource "aws_launch_template" "play_k8s_apps" {
  name_prefix             = "play-k8s-apps-"
  image_id                = "ami-04bf865d65956a33a"
  disable_api_termination = false
  disable_api_stop        = false
  update_default_version  = true

   block_device_mappings {
    device_name = "/dev/xvda"
    ebs {
      volume_size = 30  # size in GiB
      volume_type = "gp2"
      delete_on_termination = true
    }
  }

  tag_specifications {
    resource_type = "instance"
    tags = {
      "Environment" = "play"
      "Name"        = "play-k8s-apps"
    }
  }

  tag_specifications {
    resource_type = "network-interface"
    tags = {
      "Environment" = "play"
      "Name"        = "play-k8s-apps"
    }
  }

  tag_specifications {
    resource_type = "volume"
    tags = {
      "Environment" = "play"
      "Name"        = "play-k8s-apps"
    }
  }

  metadata_options {
    http_endpoint               = "enabled"
    http_protocol_ipv6          = "disabled"
    http_put_response_hop_limit = 2
    http_tokens                 = "required"
    instance_metadata_tags      = "disabled"
  }

  monitoring {
    enabled = true
  }

  vpc_security_group_ids = [
    "sg-043d6a926f1686b35"
  ]

  tags_all = {
    "Environment" = "play"
  }
}

Also my AWS EKS cluster:

module "smg_auto_play_eks" {
  source    = "terraform-aws-modules/eks/aws"
  version   = "19.15.3"

  cluster_name    = "smg-auto-play"
  cluster_version = "1.25"

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = true

  create_kms_key = false

  cluster_encryption_config = {
      provider_key_arn = aws_kms_key.smg_auto_play_eks.arn
      resources        = ["secrets"]
  }

  vpc_id     = module.smg_auto_play_vpc.vpc_id
  subnet_ids = module.smg_auto_play_vpc.private_subnets

  node_security_group_additional_rules = {
    ingress_self_all = {
      description = "Node to node all ports/protocols"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      self        = true
    }

    egress_all = {
      description      = "Node all egress"
      protocol         = "-1"
      from_port        = 0
      to_port          = 0
      type             = "egress"
      cidr_blocks      = [var.default_route]
      ipv6_cidr_blocks = ["::/0"]
    }
  }

  eks_managed_node_groups = {

    play-k8s-apps-2 = {
      use_custom_launch_template = true
       launch_template = {
         id      = aws_launch_template.play_k8s_apps.id
         version = aws_launch_template.play_k8s_apps.latest_version
     }
      min_size       = 2
      max_size       = 2
      desired_size   = 2
      instance_types = ["t3.small"]

    }
  }

  # aws-auth configmap
  manage_aws_auth_configmap = true

  aws_auth_roles = [
    {
      rolearn  = "arn:aws:iam::994442530941:role/AWSReservedSSO_AWSAdministratorAccess_7943eed9a8407aaa"
      username = "AWSReservedSSO_AWSAdministratorAccess_7943eed9a8407aaa"
      groups   = ["system:masters"]
    },
    {
      rolearn  = "arn:aws:iam::994442530941:role/AWSReservedSSO_AutomotiveplayDeveloper_04492fefd2dc7279"
      username = "AWSReservedSSO_AutomotivePlayDeveloper_04492fefd2dc7279"
      groups   = ["developers"]
    }
  ]

  aws_auth_users = [
    {
      userarn  = "arn:aws:iam::994442530941:user/ga-terraform-access"
      username = "ga-terraform-access"
      groups   = ["system:masters"]
    }
  ]

  tags = {
    Environment = var.environment
  }
}

But it creates only resources with tags, not instances with 30gb, instead with default one with 20gb. Is there some solution ?

liviudm commented 1 year ago

I'm having the same issue using version 19.15.3

Relevant parts:

module "eks" {
  source  = "registry.terraform.io/terraform-aws-modules/eks/aws"
  version = "19.15.3"

  ...

  eks_managed_node_groups = {
    karpenter = {
      block_device_mappings = {
        xvda = {
          volume_size           = 75
          volume_type           = "gp3"
          iops                  = 3000
          throughput            = 125
          encrypted             = true
          kms_key_id            = aws_kms_key.ebs.arn
          delete_on_termination = true

  ...
knkcni commented 1 year ago

Your block_device_mappings should be working with this approach :

...
  eks_managed_node_group_defaults = {
    block_device_mappings = {
      xvda = {
        device_name = "/dev/xvda"
        ebs = {
          volume_type = "gp3"
          volume_size = 100
        }
      }
    }
  }
...
joaocc commented 1 year ago

I had a similar issue, and this does indeed solve the issue. Maybe it's worth changing the defaults processing code here (https://github.com/terraform-aws-modules/terraform-aws-eks/blob/666603b6e531140d5d8fbd777cd90a7fbb8247dd/modules/eks-managed-node-group/main.tf#L43) to not accept null by default, as it will cause the provider to complain. If device_name is required, then it's easier to avoid confusion from the tf users' point of view.

github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

bryantbiggs commented 7 months ago

just coming back around to this - so I don't think there is anything we can do here

The device_name is optional - but what I suspect folks are running into is that they want the root volume to be changed based on these configs provided. In order for that to take place, you need to match the device name in your config with the device name of the AMI's root volume

However this will vary, and some AMIs have multiple volumes (i.e. - Bottlerocket has a root volume and a separate data volume on the AMI they provided). So with that, I don't see any code changes that we can or will make here for this, but if I am missing something, please let me know

github-actions[bot] commented 6 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.