search

terraform-aws-modules / terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
Apache License 2.0
4.45k stars 4.07k forks source link

SSH access combined with custom user_data #2610

Closed florian-besser closed 1 year ago

florian-besser commented 1 year ago

Description

I tried to do something stupid, namely combine the example given for remote_access with the examples for enable_bootstrap_user_data / pre_bootstrap_user_data. I expected these things to work together as I wanted to create a node group to which I could a) connect via SSH and b) could pass custom kubelet settings

Versions

Reproduction Code [Required]

resource "tls_private_key" "worker_node_access" {
  algorithm = "RSA"
}

resource "aws_key_pair" "worker_node_access" {
  key_name_prefix = "worker_node_access"
  public_key      = tls_private_key.worker_node_access.public_key_openssh
}

data "aws_ami" "eks_default" {
  most_recent = true
  owners      = ["amazon"]

  filter {
    name   = "name"
    values = ["amazon-eks-node-1.25-v*"]
  }
}

module "eks" {
  source  = "registry.terraform.io/terraform-aws-modules/eks/aws"
  version = "18.31.2"

  cluster_name    = "foo"
  cluster_version = "1.25"

  eks_managed_node_group_defaults = {
    disk_size              = 20
    instance_types         = ["t3.medium"]
  }
  eks_managed_node_groups = {
    foo = {
      min_size                   = 1
      max_size                   = 15
      desired_size               = 5
      ami_type                   = "CUSTOM"
      ami_id                     = data.aws_ami.eks_default.image_id
      enable_bootstrap_user_data = true
      # By default, the module creates a launch template to ensure tags are propagated to instances, etc.,
      # so we need to disable it to use the default template provided by the AWS EKS managed node group service
      create_launch_template = false
      launch_template_name   = ""
      # Remote access cannot be specified with a launch template
      remote_access          = {
        ec2_ssh_key               = aws_key_pair.worker_node_access.key_name
      }
      bootstrap_extra_args = "--kubelet-extra-args '--max-pods=20'"
    }
}

Steps to reproduce the behavior:

Create an EKS managed node group with the following:

For my example I followed https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/user_data.md#eks-managed-node-group for bootstrap_extra_args. I followed https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/examples/eks_managed_node_group/main.tf for ssh_access

Expected behavior

Clear and concise error message that specifying create_launch_template=false / launch_template_name = "" are incompatible with bootstrap_extra_args & similar arguments. OR: Both settings applied, max-pods should now be 20, while SSH access works.

Actual behavior

The node group is created without any customer user_data being applied The SSH access works very nicely. max-pods is 17, the default value AWS EKS applies it seems.

Terminal Output Screenshot(s)

Applies successfully

Additional context

It would be really nice if we could get an example how to manage SSH access to nodes without disabling the launch template so we can still modify user_data.

florian-besser commented 1 year ago

After wrangling with this problem for a bit I used the following solution:

eks_managed_node_group_defaults = {
    disk_size = 20
    instance_types = ["t3.medium"]
  }
  eks_managed_node_groups = {
    foo = {
      min_size     = 1
      max_size     = 15
      desired_size = 5

      ami_type                 = "CUSTOM"
      ami_id                   = data.aws_ami.eks_default.image_id
      key_name = aws_key_pair.worker_node_access.key_name
      enable_bootstrap_user_data = true
      bootstrap_extra_args = "--kubelet-extra-args '--kube-reserved memory=0.2Gi,ephemeral-storage=1Gi --system-reserved memory=0.1Gi,ephemeral-storage=1Gi --eviction-hard memory.available<200Mi,nodefs.available<10%'"
    }
  }

Basically I removed remote_access and replaced it with key_name. I also removed create_launch_template=false / launch_template_name = "". The effect of that we're now again generating a launch template (with which we can use user_data). This solves one part, but now we lost the SSH access. By adding key_name we get SSH access back, and this time the key pair is persisted in the launch template, meaning we now have a launch template with custom arguments and SSH.

Maybe this could be added as an example when using remote_access isn't permissible?

bryantbiggs commented 1 year ago

I am hesitant to do do anything for this - its down to understanding how the service handles these properties and its not down to the module itself. The module just enables you to use the service

If you use the default launch template provided by EKS Managed nodegroups - you can use remote access and disk size values but you can't set other values that are only available when using a custom launch template, and vice versa. The proper solution here, in my opinion, is to use SSM and not use SSH access

github-actions[bot] commented 1 year ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.