Add LaunchTemplate SSH KeyName to AWSNodeTemplate

[ksquaredkey]

Tell us about your request In migrating a load from a Terraform EKS self_managed_node_group to Karpenter, I cannot find how to set the EC2 key_name to use for the Instance.

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Looks like we just need a way to take KeyName at the AWSNodeTemplate and pass through to the Karpenter-built LaunchTemplate. I can create my own LaunchTemplate to set the KeyName, but that seems like overkill and a maintenance headache when the rest of the Karpenter-built LaunchTemplate is so close to what I need. I know that SSH'ing into a Node is an anti-pattern, but sometimes we need to do this to debug in our dev environment ;-)

Are you currently working around this issue? Still looking through my options. My go is weak, but I might try to add myself and make a PR. However, don't want to duplicate effort if someone else is already on this. Or if this is deliberately missing. I didn't see any Issues are PRs that covered this, though.

Additional context

Attachments

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[FernandoMiguel]

it's strongly recommend you use aws ssm manager instead of SSH https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html

[bwagner5]

As @FernandoMiguel we recommend using SSM rather than static SSH keys. SSM will allow you to login to the instance in an SSH-like session. The other option which uses dynamic SSH keys is ec2-instance-connect, which is another option if you need real SSH session, but there's no need to provide any keys up-front and works great with karpenter.

[ksquaredkey]

Yes, I am familiar of SSM.

Sadly, it is not capable of the features I need to work with my large company's existing development infrastructure that has been built up based on the bog-standard old-time features of EC2 and EKS - namely, the SSH keypair. One in particular - I have yet to find ad-hoc port forwarding over the SSM session back to the client machine.

All I am trying to do is to make the Karpenter-created nodes look as close to our existing Terraform EKS created self-managed nodes as possible, using a bog-standard feature of EC2 and EKS. I'm glad you folks are able to leverage SSM for your needs, but one size does not fit all.

[ksquaredkey]

I'll look into ec2-instance-connect - that is a new one on me.

[ksquaredkey]

Ah, ec2-instance-connect has another similar problem as SSM. Not everyone that has an occasional need to access at the OS level nodes in our VPC that is connected gets to have AWS accounts. There is a need for authentication outside of the IAM/AWS SSO world. I'm glad for you that you do not have it, but some of us do.

[ksquaredkey]

I was able to work around the missing keyName feature by leveraging the userData. Replace the "{{ insertFile..." with your keys there.

` userData: | MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="BOUNDARY"

--BOUNDARY
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
mkdir -p ~ec2-user/.ssh/
touch ~ec2-user/.ssh/authorized_keys
cat >> ~ec2-user/.ssh/authorized_keys <<EOF

{{ insertFile "../../../../docker-eks/terraform/keys/master-authorized_keys" | indent 4 }} EOF chmod -R go-w ~ec2-user/.ssh/authorized_keys chown -R ec2-user ~ec2-user/.ssh

--BOUNDARY--

`

[chrisnegus]

The information about adding SSH credentials via userData (see above) was added to the docs in PR #2500. So I'm going to close this issue.

[armenr]

@chrisnegus - As a former Amazonian, I find it bizarre that the Karpenter team would presume to make the decision on behalf of ALL AWS customers as to whether customers are essentially "allowed" to configure EC2 instances with SSH keys or not.

That's insane to me.

In my case, I am using a piece of software which expects an EC2 instance to provide data on an SSH key through the instance metadata.

Because the authors/maintainers made some bad choices themselves, the software in question actually outright fails if it's operating on/running on an EC2 instance without a designated SSH key. It poops its pants and exits gracelessly if there's no IMDS data returned for the EC2 instance's SSH public key.

And even MORE coincidentally, the software in question is a lightweight drop-in replacement for cloud-init. SO, I cannot run userdata on my EC2, since it crashes when it finds that IMDS doesn't return any info on the SSH key for the host...

...Which means that the workaround instructions included in this PR won't help, at all...because it assumes that being able to shim your own authorized SSH keys through userdata is a sufficient or appropriate "fix."

So, I'm sitting here with months of work I've done, blocked, because the Karpenter team essentially wants to force what is considered a "best practice" on users and customers as a de-facto reality, with no way to opt in or opt out.

This is kinda crazy to me. How hard would it be to simply pass through the key/value pair (as optional, with a default set to "nope, no SSH keys please") through your API?

[ksquaredkey]

Glad it isn't just me. After the reception above, I abandoned the idea of working on a PR. I sure sounded like a simple bit of code that my early Golang coding could handle as an exercise. However, I didn't want to put in the time if there was a good chance of flat-out rejection of the effort (I still consider it a bug, but I've got other windmills to tilt).

[armenr]

I sounded a bit...strong in my last comment, but the point I really just want to make is this:

Customers/Builders will tend to expect/assume/depend - quite naturally - that the Karpenter NodeTemplate API allow them to do and to configure all of the same things that they've always been able to do and to configure (through the AWS Console, through the AWS EC2 APIs, through CloudFormation...etc).

And to bury the fact that Karpenter enforces this kind of opinionation (e.g. "You cannot SSH these hosts, we don't 'support' it") is unfair to so many customers who might be re-platforming or re-architecting around EKS + Karpenter, not knowing (for example) that they can't provide their Ops/SRE/Dev teams with something as fundamental as SSH.

Granted, it is a best-practice to use the SSM Agent instead, and that you can ALSO use SSM-Agent + SSH + an SSM Agent Policy Document to access EC2 instances that don't have public IPs...but to preclude the ability of any team to choose for themselves whether they want SSH on their Nodes or not is a bit of a reach.

[ellistarn]

We just discussed this as a team. Given how contained this feature is, and the fact that it's a subset of launch template API surface, we'd be happy to accept a PR that wires this through.

[edgan]

Granted, it is a best-practice to use the SSM Agent instead, and that you can ALSO use SSM-Agent + SSH + an SSM Agent Policy Document to access EC2 instances that don't have public IPs...but to preclude the ability of any team to choose for themselves whether they want SSH on their Nodes or not is a bit of a reach.

I don't even agree, given the details, that SSM Agent is best-practice. If you are using Ubuntu the SSM Agent is a snap, and I don't want snaps. Yes, this is more of a Ubuntu-ism, but it is also reality.

[armenr]

@edgan - I wouldn't install any snaps on my Ubuntu hosts either. I totally get what you mean.

[infa-ddeore]

looks like no progress on this issue yet, any updates on it?

[matschaffer-roblox]

As far as I can tell the only way to copy data directly to/from an ec2 instance requires an ssh key, even when using session manager or ec2 instance connect.

The user data approach in https://karpenter.sh/docs/faq/#can-i-add-ssh-keys-to-a-nodepool is a bit sub-optimal because you lose the ability to leverage the KeyName from the EC2 API. You would either have to assume a key, or implement your own key information via instance tags.

Given that, I'm glad to see @ellistarn's note that PRs will be accepted to properly support KeyName.