Template Known Karpenter Values into Custom AMIFamily UserData

[yogeek]

Description

Observed Behavior:

As explained in this Slack thread, I have an error when I do not specify the clusterEndpoint argument when installing karpenter

{"level":"FATAL","time":"2023-11-03T16:41:24.082Z","logger":"controller","message":"unable to detect the cluster endpoint, failed to resolve cluster endpoint,

The "clusterEndpoint" seems to be mandatory but in a non-EKS case, I do not see the use of such a parameter.

What exactly is the role of this argument please ?

If it is used for the node to be able to join the cluster, in my case, all the elements to do the "join" are all included in the user-data script that I specify in the nodepool.

If it is for karpenter need to communicate with the API server, as Karpenter is already inside the cluster, I imagined it would be able to talk the the API server through the internal endpoint (https://kubernetes.default.svc/) without needing the external endpoint URL...

Expected Behavior:

I expected the "clusterEndpoint" to be optional.

Versions:

• Chart Version: 0.32.1
• Kubernetes Version: 1.25

[jonathan-innis]

Right now, the requirement is driven by the fact that we auto-generate the userData when we are creating EC2 nodes and every node that is joining the cluster needs to know that endpoint in one way or another to do an initial reach-out. I'm assuming that you need this endpoint as well in your userData for the nodes to join the cluster, even if you have some custom join procedure? Is the difference just that you specify this value directly in your userData rather than at the Karpenter level?

Would it make sense to have it required if we allowed you to template your userData for the Custom AMIFamily in a way that allowed you to specify a template value for the cluster endpoint, like {{ .ClusterEndpoint }}

[yogeek]

ok I get it. I took a look at the code and indeed, in the case of a "custom" AMI Family, the options specified in the nodepool.yaml are currently ignored apparently (https://github.com/aws/karpenter/blob/main/pkg/providers/amifamily/bootstrap/custom.go) whereas in the case of other AMI Family all these options are passed to build various elements (kubelet extra args, labels, taints...) like we can see here : https://github.com/aws/karpenter/blob/main/pkg/providers/amifamily/bootstrap/bootstrap.go

In our case, we are not having directly the clusterEndpoint in our user-data script : the user-data script of our nodes downloads the kubeconfig from an s3 bucket (this kubeconfig was previously uploads to the s3 bucket by the user-data of the first control-plane) and the kubeadm join command is launched with this kubeconfig. And this is this kubeconfig that contains the clusterEndpoint (API server URL)

I think the issue here is larger than only the "clusterEndpoint" field in fact.

I am currently having a slack conversation about the non-propagation of labels (cf. https://kubernetes.slack.com/archives/C02SFFZSA2K/p1700592018121909 and I just saw that it is also with you @jonathan-innis :smile: ) : don't you think both issues have in fact the same root cause ?

It seems to me that the "Custom" AMIFamily does not work correctly with today's implementation because it is not capable of passing YAML fields of the nodepool spec to the nodes (kubelet, labels, annotations, ...) and maybe it is also because of that that other information like the providerId is not propagated from the nodeClaim to the node

So indeed, we need to have some way of passing information from the nodepool yaml spec to the node manifest in the case of the custom AMIFamily...

What do you think ? Do you already have some working example of custom AMIFamily that worked successfully ?

[jonathan-innis]

What do you think ? Do you already have some working example of custom AMIFamily that worked successfully

Yeah, the whole reason that the Custom AMIFamily exists is exactly because we can't know the process that you are using to configure the kubelet that is eventually going to reach out to a control-plane that we also don't know the configuration of (it may be an EKS control plane, but we also support other control planes that aren't EKS).

Realistically, once you get into the Custom AMIFamily territory, it's more on the user to configure the userData in a way that works appropriately for the node to join and Karpenter (and other control-plane controllers like cloud-controller-manager) to be able to properly track the new node. I think the template solution that I mentioned above is about as close as we can get to providing all of the data that Karpenter knows about to be able to pass in the relevant information into the user data.

As for Karpenter passing in the provider id to the user data itself, this isn't possible for Karpenter to do since the instance hasn't launched yet when Karpenter is configuring the user data. Ultimately, it needs to be the responsibility of the user data after the instance is running to go to IMDS to look-up and set its provider id.

[yogeek]

ok makes sense, thanks for the clarification.

I guess the template solution you proposed would be a nice improvement then in order to benefit from both the Custom AMIFamily (by keeping control over our user-data) and the karpenter YAML specification that allows to manage some fields declaratively in order to have the karpenter reconciliation process