How do I set HTTP proxy for Amazon EKS containerd nodes?

c-s-u commented 1 year ago

[x] ✋ I have searched the open/closed issues and my issue is not listed.

Please describe your question here

I would like to set up an https proxy for the nodes of my fully-private EKS cluster. How can I implement like the link below in terraform? https://repost.aws/knowledge-center/eks-http-proxy-containerd-automation

Provide a link to the example/module related to the question

Additional context

best regards.

askulkarni2 commented 1 year ago

Hi @chugai-so-ukiyama, thanks for issue. At the moment we do not have this pattern implemented in this project. I will add this to our backlog for now. In the meantime, I recommend taking a look at https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/docs/user_data.md for information on how to pass user data provided in that article to nodes of a MNG. Please feel free to post any findings/issues here.

horkko commented 1 year ago

Hi @askulkarni2

Maybe you'll be able to light my lantern ? 🤞

I've a similar demand for a managed node group running bottelrocket AMI operation system. I've tried to set, in the managed_node_groups map, a post_userdata value (generated with a templatefile as show below) but it looks like this is not taken into consideration. The user_data generated into the final launch_template does not integrate what I've passed topost_userdata input. It only contains these data:

[settings.kubernetes]
"cluster-name" = "eks-z1-mwaa-test-mng-calico"
"api-server" = "https://44B08416040A17E5F574A6E63F6738D5.gr7.eu-west-3.eks.amazonaws.com"
"cluster-certificate" = ...
"cluster-dns-ip" = "172.20.0.10"
[settings.kubernetes.node-labels] <==== created through k8s_labels variable
"lifecycle" = "on_demand"
"eks.amazonaws.com/nodegroup-image" = "ami-0e63a05ceec3cf941"
"eks.amazonaws.com/capacityType" = "ON_DEMAND"
"eks.amazonaws.com/nodegroup" = "managed-ondemand-1-20230908145012907000000005"

Am I missing something, or is it a normal behavior from the [aws-eks-managed-node-groups](https://github.com/aws-ia/terraform-aws-eks-blueprints/tree/v4.32.1/modules/aws-eks-managed-node-groups) module? Looking at the terraform code, I don't understand why this user_data is not in final launch_template ?

Sample code from module call:

module "eks_blueprints" {
  source = "github.com/aws-ia/terraform-aws-eks-blueprints?ref=v4.32.1"
...
managed_node_groups = var.managed_node_groups

}

managed_node_groups variable details:

  managed_node_groups = {

    "airflow-workers" = {
      ami_type       = "BOTTLEROCKET_x86_64"
      capacity_type  = "ON_DEMAND"
      instance_types = var.eks_mng_instance_types
      subnet_ids     = data.aws_subnets.private["z1"].ids
      max_size       = var.eks_mng_max_size
      min_size       = var.eks_mng_min_size
      desired_size   = var.eks_mng_desired_size
      node_group_name = "${var.node_group_name}-1"
      k8s_taints                  = []
      k8s_labels                  = { lifecycle = lower(var.eks_mng_capacity_type) } # var.eks_mng_k8s_labels
      enable_metadata_options     = true
      http_endpoint               = "disabled"
      http_tokens                 = "required" #tfsec:ignore:aws-autoscaling-enforce-http-token-imds
      http_put_response_hop_limit = 1
      http_protocol_ipv6          = null
      instance_metadata_tags      = "disabled"
      bootstrap_extra_args        = "--use-max-pods false --container-runtime containerd"
      kubelet_extra_args          = "--node-labels=arch=x86,WorkerType=${var.eks_mng_capacity_type} --max-pods=110"
      custom_ami_id               = data.aws_ami.mng_ami.id #"ami-0e63a05ceec3cf941"
      launch_template_os          = "bottlerocket"  <====
      create_launch_template      = true                <==== Is it set to true (https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/8a06a6e7006e4bed5630bd49c7434d76c59e0b5e/modules/aws-eks-managed-node-groups/managed-launch-templates.tf#L2)
      pre_userdata                = ""
      post_userdata               = templatefile("${path.module}/templates/eks_mng_node_postuserdata.tpl",
        {
          node_proxy    = local.node_proxy,
          node_no_proxy = local.node_no_proxy
      })
      # block_device_mappings only used when create_launch_template is set to true
      # https://github.com/aws-ia/terraform-aws-eks-blueprints/blob/v4.32.1/modules/aws-eks-managed-node-groups/managed-launch-templates.tf#L2
      block_device_mappings = [{
        device_name           = "/dev/xvda"
        volume_type           = var.eks_mng_block_volume_type #"gp3" # The volume type. Can be standard, gp2, gp3, io1, io2, sc1 or st1 (Default: gp3).
        volume_size           = 150
        delete_on_termination = true
        encrypted             = true
        kms_key_id            = var.ebs_kms_key_id
        throughput            = 125
      }]
    }
  }

template file content as describe here : https://github.com/bottlerocket-os/bottlerocket#proxy-settings

[settings.network]
https-proxy = "${node_proxy}"
no-proxy    = "${node_no_proxy}"

By the way, same issue with block_device_mappings in manage_node_groups map.

Can you help with this?

Thanks a lot for your time.

Emmanuel

aws-ia / terraform-aws-eks-blueprints