Volodymyr-Kuchinskyi commented 1 month ago

Description

Recently I've updated ami_type of eks cluster (version 1.28) nodes from default one AL2_x86_64 to AL2023_x86_64_STANDARD and noticed that eks ebs-csi-controller (ebs-attacher) was not able to attach volume to the ec2 node, and I've received such an error:

AttachVolume.Attach failed for volume "pvc-uuid" : rpc error: code = Internal desc = Could not attach volume "vol-<volume-id>" to node "i-<node-id>": error listing AWS instances: operation error EC2: DescribeInstances, get identity: get credentials: failed to refresh cached credentials, no EC2 IMDS role found, operation error ec2imds: GetMetadata, failed to get rate limit token, retry quota exceeded, 0 available, 5 requested

When I've checked the new launch template metadata options, i've noticed that HttpPutResponseHopLimit was set to 1 by default.However according to eks best practices it should be set to 2 if you need an access to IMDSv2 from the application pod itself, and for old AMI type AL2_x86_64 it indeed was set to 2 by default, so when I switched back node group with deployed ebs-csi-controller to AL2_x86_64, everything started to work properly again.

It would be really helpful to specify all the metadata_options for aws_eks_node_group resource itself for those who do not want to support their own launch_template for eks nodes.

Affected Resource(s) and/or Data Source(s)

aws_eks_node_group

Potential Terraform Configuration

resource "aws_eks_node_group" "upland_cluster_node_groups" {
  cluster_name    = "eks-cluster"
  node_group_name = "eks-cluster-node-group-1"
  version         = 1.28
  node_role_arn   = ...
  subnet_ids      = ...
  instance_types  = ["m5.xlarge"]
  capacity_type   = "ON_DEMAND"
  ami_type        = "AL2023_x86_64_STANDARD"
  disk_size       = 50

  instance_metadata_options = {
    http_endpoint               = "enabled"
    http_tokens                 = "required"
    http_put_response_hop_limit = 2
    instance_metadata_tags      = "enabled"
  }

  labels = {
    "node-group" = "node-group-1"
  }

  scaling_config {
    min_size     = 1
    max_size     = 3
    desired_size = 2
  }

  depends_on = [ ... ]
}

References

https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest/submodules/eks-managed-node-group#input_metadata_options

https://github.com/cloudposse/terraform-aws-eks-node-group/blob/main/README.md#input_metadata_http_put_response_hop_limit

Would you like to implement a fix?

None

github-actions[bot] commented 1 month ago

Community Note

Voting for Prioritization

Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
Please see our prioritization guide for information on how we prioritize.
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

If you are interested in working on this issue, please leave a comment.
If this would be your first contribution, please review the contribution guide.

eggshell commented 1 week ago

I would also like this feature.

Ishwarya-23 commented 1 week ago

Yes this would be an useful feature.

bryantbiggs commented 2 days ago

this is not controlled by Terraform - this is due to AL2023. From the EKS AMI release note on AL2023 support:

AL2023 requires IMDSv2 by default. IMDSv2 has several benefits that help improve security posture. It uses a session-oriented authentication method that requires the creation of a secret token in a simple HTTP PUT request to start the session. A session's token can be valid for anywhere between 1 second and 6 hours. For more information on how to transition from IMDSv1 to IMDSv2, see Transition to using Instance Metadata Service Version 2 and Get the full benefits of IMDSv2 and disable IMDSv1 across your AWS infrastructure. If you would like to use IMDSv1, you can still do so by manually overriding the settings using instance metadata option launch properties. For IMDSv2, the default hop count for managed node groups is set to 1. This means that containers won't have access to the node's credentials using IMDS. If you require container access to the node's credentials, you can still do so by manually overriding the HttpPutResponseHopLimit in a custom EC2 launch template, increasing it to 2, and by using EKS Pod Identity.

If you wish to change this, you will need to provide a custom launch template to your node group

justinretzolk commented 2 days ago

With the above in mind, there's nothing for the AWS Provider team to do, so I'm going to close this issue. If you have additional requests in the future, please do let us know!

github-actions[bot] commented 2 days ago

[!WARNING] This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.

hashicorp / terraform-provider-aws