eks 1.22

[csmith-simplebet]

Has anyone experienced any issues with deploying kube2iam to a fresh install of eks 1.22? I am getting the following error: "Error getting instance id, got status: 401 Unauthorized"

My role permissions and trusted relationships are right. Im running this the same way in three other environments on eks 1.19 with no issues.

Thanks.

[pd-kcarrasco]

@csmith-simplebet Could be due to the the move to BoundServiceAccountTokenVolume which are service account "tokens that are audience, time, and key bound". These are set to expire every 90 days on EKS. See more on the EKS kubernetes version page for 1.22. There is an important warming explaining this. You should see a similar warning on the AWS web console for your EKS cluster.

Although the warning talks about updating your apps to use the latest Kubernetes client SDKs (not the latest AWS SDKs), the version of the aws-sdk-go used in this project is from 03 Apr 2017. Maybe that version doesn't handle token refreshes yet?

[davidegiunchi]

@csmith-simplebet Could be due to the the move to BoundServiceAccountTokenVolume which are service account "tokens that are audience, time, and key bound". These are set to expire every 90 days on EKS. See more on the EKS kubernetes version page for 1.22. There is an important warming explaining this. You should see a similar warning on the AWS web console for your EKS cluster.

Although the warning talks about updating your apps to use the latest Kubernetes client SDKs (not the latest AWS SDKs), the version of the aws-sdk-go used in this project is from 03 Apr 2017. Maybe that version doesn't handle token refreshes yet?

@pd-kcarrasco Since i was getting scary, i've done a little research: i don't think that this is the problem. Looking at the issue that you linked BoundServiceAccountTokenVolume its written:

The following Kubernetes client SDKs refresh tokens automatically within the required time frame:

    Go v0.15.7 and later

AWS is writing about kubernetes SDK, not aws-sdk, then looking at the kube2iam code , it's installed the 0.17.3 version, that's more recent than the required 0.15.7. Another thing is that this kubernetes requirement is made even on kubernetes 1.21, and i'm running it since 1 month without any problem.

[friedrich-brunzema]

@csmith-simplebet Ran into the same issue a while back.

This is related to the introspection api having security by default now. By default, EC2 uses IMDSv2, and the nodes have to be told to run without it. I use the terraform-aws-eks module to create the cluster -- there you have to set

  eks_managed_node_group_defaults = {
    ami_type        = "AL2_x86_64"
    instance_types  = ["m5n.xlarge"]
    platform        = "linux"
    use_name_prefix = true
    update_config = {
      max_unavailable_percentage = 10
    }
    attach_cluster_primary_security_group = true

    # this is critical for kube2iam to work without IMDSv2, ie otherwise
    # the instance does NOT have access to http://169.254.169.254/latest/meta-data/
    metadata_options = {
      http_endpoint               = "enabled"
      http_tokens                 = "optional"
      http_put_response_hop_limit = 2
    }

You can also set the metadata_options block in the eks_managed_node_groups

eks_managed_node_groups = {
    tooling = {
      metadata_options = {
        http_endpoint               = "enabled"
        http_tokens                 = "optional"
        http_put_response_hop_limit = 2
      }
      name             = "eks-tooling"
      subnet_ids       = dependency.vpc.outputs.private_subnets
      min_size         = 1
      max_size         = 10
      desired_capacity = 1
      instance_types   = ["m5n.xlarge"]
      capacity_type    = "ON_DEMAND"
      labels = {
        eks_namespace = "tooling"
      }
    },

[Insidexa]

use IMDSv1 version - that's worked for me