search

terraform-aws-modules / terraform-aws-eks

Terraform module to create Amazon Elastic Kubernetes (EKS) resources 🇺🇦
https://registry.terraform.io/modules/terraform-aws-modules/eks/aws
Apache License 2.0
4.47k stars 4.08k forks source link

It seems to be the self-managed node group does not support IPv6 Cluster #2805

Closed vkaylee closed 1 year ago

vkaylee commented 1 year ago

Description

Hello guys, I have found the issue with features: IPv6 and self_managed_node_groups After provisioning the ipv6 cluster with a self-managed node group, the taint appeared node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule with node status: Ready

Everything has been checked, it has some differences:

--// Content-Type: text/x-shellscript; charset="us-ascii"

!/bin/bash

set -ex B64_CLUSTER_CA= API_SERVER_URL=https://.gr7.ap-southeast-1.eks.amazonaws.com K8S_CLUSTER_DNS_IP=fd33:75f9:08f7::a /etc/eks/bootstrap.sh eks-nA5EE1ws --kubelet-extra-args '--node-labels=eks.amazonaws.com/sourceLaunchTemplateVersion=1,eks.amazonaws.com/nodegroup-image=ami-0ef7bbe64818b655d,eks.amazonaws.com/capacityType=SPOT,eks.amazonaws.com/nodegroup=eks--green-...,eks.amazonaws.com/sourceLaunchTemplateId=lt- --max-pods=98' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --ip-family ipv6 --service-ipv6-cidr fd33:75f9:08f7::/108 --use-max-pods false

--//--


Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration (see the `examples/*` directory for references that you can copy+paste and tailor to match your configs if you are unable to copy your exact configuration). The reproduction MUST be executable by running `terraform init && terraform apply` without any further changes.

If your request is for a new feature, please use the `Feature request` template.

- [x] ✋ I have searched the open/closed issues and my issue is not listed.

## ⚠️ Note

Before you submit an issue, please perform the following first:

1. Remove the local `.terraform` directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): `rm -rf .terraform/`
2. Re-initialize the project root to pull down modules: `terraform init`
3. Re-attempt your terraform plan or apply and check if the issue still persists

## Versions

- Module version [Required]:
19.16

- Terraform version:
 v1.6.2
<!-- Execute terraform -version -->
- Provider version(s):
<!-- Execute: terraform providers -version -->
    + provider registry.terraform.io/hashicorp/aws v5.23.1
    + provider registry.terraform.io/hashicorp/cloudinit v2.3.2
    + provider registry.terraform.io/hashicorp/helm v2.11.0
    + provider registry.terraform.io/hashicorp/kubernetes v2.23.0
    + provider registry.terraform.io/hashicorp/local v2.4.0
    + provider registry.terraform.io/hashicorp/null v3.2.1
    + provider registry.terraform.io/hashicorp/random v3.5.1
    + provider registry.terraform.io/hashicorp/time v0.9.1
    + provider registry.terraform.io/hashicorp/tls v4.0.4

## Reproduction Code [Required]

<!-- REQUIRED -->
```terraform
provider "aws" {
  region = local.region
}

provider "kubernetes" {
  host                   = module.eks.cluster_endpoint
  cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

  exec {
    api_version = "client.authentication.k8s.io/v1beta1"
    command     = "aws"
    # This requires the awscli to be installed locally where Terraform is executed
    args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
  }
}

provider "helm" {
  kubernetes {
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)

    exec {
      api_version = "client.authentication.k8s.io/v1beta1"
      command     = "aws"
      # This requires the awscli to be installed locally where Terraform is executed
      args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
    }
  }
}

data "aws_availability_zones" "available" {}

locals {
  name   = basename(path.cwd)
  region = "us-west-2"

  vpc_cidr = "10.0.0.0/16"
  azs      = slice(data.aws_availability_zones.available.names, 0, 3)

  tags = {
    Blueprint  = local.name
    GithubRepo = "github.com/aws-ia/terraform-aws-eks-blueprints"
  }
}

################################################################################
# Cluster
################################################################################

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 19.16"

  cluster_name                   = local.name
  cluster_version                = "1.27"
  cluster_endpoint_public_access = true

  # IPV6
  cluster_ip_family          = "ipv6"
  create_cni_ipv6_iam_policy = true

  cluster_addons = {
    coredns    = {}
    kube-proxy = {}
    vpc-cni    = {}
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets

  manage_aws_auth_configmap = true

  eks_managed_node_groups = {
    initial = {
      instance_types = ["m5.large"]

      min_size     = 1
      max_size     = 3
      desired_size = 2
    }
  }

  self_managed_node_groups = {
    mixed = {
      name = "mixed"

      min_size     = 1
      max_size     = 5
      desired_size = 2

      bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=spot'"

      use_mixed_instances_policy = true
      mixed_instances_policy = {
        instances_distribution = {
          on_demand_base_capacity                  = 0
          on_demand_percentage_above_base_capacity = 20
          spot_allocation_strategy                 = "capacity-optimized"
        }

        override = [
          {
            instance_type     = "m5.large"
            weighted_capacity = "1"
          },
          {
            instance_type     = "m6i.large"
            weighted_capacity = "2"
          },
        ]
      }
    }
  }

  tags = local.tags
}

################################################################################
# Supporting Resources
################################################################################

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "~> 5.0"

  name = local.name
  cidr = local.vpc_cidr

  azs             = local.azs
  private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
  public_subnets  = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]

  enable_nat_gateway     = true
  single_nat_gateway     = true
  enable_ipv6            = true
  create_egress_only_igw = true

  public_subnet_ipv6_prefixes                    = [0, 1, 2]
  public_subnet_assign_ipv6_address_on_creation  = true
  private_subnet_ipv6_prefixes                   = [3, 4, 5]
  private_subnet_assign_ipv6_address_on_creation = true

  public_subnet_tags = {
    "kubernetes.io/role/elb" = 1
  }

  private_subnet_tags = {
    "kubernetes.io/role/internal-elb" = 1
  }

  tags = local.tags
}

Steps to reproduce the behavior:

Expected behavior

Self-managed node group has:

Actual behavior

Self-managed node group has:

Terminal Output Screenshot(s)

Additional context

bryantbiggs commented 1 year ago

There isn't anything related to self-managed nodegroups that would restrict IPv6 usage - however, you do need to specify the correct bootstrap arguments for IPv6 https://github.com/awslabs/amazon-eks-ami/blob/e99201099a3c4752af465c03bd5dedda0a729ca8/files/bootstrap.sh#L33C14-L33C20

github-actions[bot] commented 11 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.