search

aws / karpenter-provider-aws

Karpenter is a Kubernetes Node Autoscaler built for flexibility, performance, and simplicity.
https://karpenter.sh
Apache License 2.0
6.89k stars 970 forks source link

httpPutResponseHopLimit: 1 not compatible with EBS csi driver #7200

Open drawnwren opened 1 month ago

drawnwren commented 1 month ago

Description

Karpenter defaults to httpPutResponseLimit: 1, but according to this (rather hard to find) ebs csi documentation, https://github.com/kubernetes-sigs/aws-ebs-csi-driver/blob/master/docs/install.md#imds-ec2-metadata, httpPutResponseLimit: 2 is required for EBS compatibility. 2 might be a default that makes more sense (give that I think ebs is a fairly common add-on to use?).

engedaam commented 1 month ago

The team wanted to take in security best practice by Disabling IMDS Access from Containers by Default. However, I do think this should be called out in our documentation to give some guidance to customers

engedaam commented 1 month ago

@drawnwren Why are not able to use Kubernetes Metadata? Seems like that would a reasonable fallback

If the driver is unable to reach IMDS, it will fallback to using the Kubernetes API. For this metadata source to work, the driver pods must have access to the Kubernetes API server. Additionally, the Kubernetes node objects must include the following information:

  • Instance ID (in the Node's ProviderID)
  • Instance Type (in the label node.kubernetes.io/instance-type)
    • Instance Region (in the label topology.kubernetes.io/region)
  • Instance AZ (in the label topology.kubernetes.io/zone)
drawnwren commented 1 month ago

I'm not sure. I'm just using the default eks method for adding the ebs-csi-controller to the cluster (through the aws dashboard) and it seems like it doesn't actually fallback to kubernetes metadata or there's something on the cluster that I would need to configure for this to work but I'm not aware of what it would be?

engedaam commented 1 month ago

These values are typically set by the AWS CCM. You must have the AWS CCM or a similar tool installed in your cluster providing these values for Kubernetes metadata to function.

Are you using AWS CCM in your cluster?