Support for User Data from Secrets

[isugimpy]

Description

What problem are you trying to solve? When running a self-managed kubeadm cluster, a common way to bootstrap the join of the nodes is via the TLS bootstrap mechanism, which requires passing a shared token from worker nodes to the control plane. This token should generally be considered a secret, because it can allow arbitrary nodes to join a cluster. Unfortunately, when passing user data to create nodes via Karpenter, it appears there's no mechanism to secure or obfuscate this secret data. While it is possible to retrieve the token from an external source at runtime on the nodes, it may be helpful and desirable to instead store the complete text of the user data in a kind: Secret so Karpenter can read it and inject at runtime, instead of storing the full text inline on the NodeTemplate.

How important is this feature to you? Moderately important. It would ease some complexities. Not critical. Open to writing this myself if it might be accepted and there's not someone else who might get to it first.

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[njtran]

Generally it looks like the EC2 Best Practices recommendation is to use another stored mechanism that your userData references to get these secrets.

Generally, if we accepted this, it would probably look like either making the userData a kind:Secret that can be referenced, so that users can restrict read permissions to only the IAM users that need it.

Is there a reason you need to include the secrets in your userData? Can you not drive this through AWS Secrets Manager or Parameter Store that your instance would have access to with the instanceRole?

[isugimpy]

There's no strict need to include it in the userData. This is more a convenience option. I could also see it feasibly being used for sourcing from a ConfigMap for non-secret things, to allow a given userData blob to be reused by multiple NodeTemplates without duplicating across them.

[njtran]

I see, so generally you're looking for a EnvFrom type behavior, so that nodes could reference data in a configmap for the bootstrapping process. Is that right?

[isugimpy]

Yeah, that's a good way to put it.

[jonathan-innis]

Open to writing this myself if it might be accepted and there's not someone else who might get to it first

I'm not necessarily strictly opposed to this; however, I think we've generally taken a stance that we want to try to reduce the configuration surface in Karpenter and allow users to template out the surface that is actually propagated down to the raw CustomResources through something like Terraform, Kustomize, Helm, etc.

I think we're sort of looking at two separate use-cases here just to clarify:

1. The ability to specify secret values and put them into your userData a. I think we are generally saying that this is a bad idea and not something that we would recommend as part of the EC2 Best Practices.
2. The ability to re-use userData across AWSNodeTemplates a. This one is interesting but I'm not sure that we've seen enough utility for this use-case to support it natively through the AWSNodeTemplate interface.