deprecation/removal of ssh-rsa from PubKeyAcceptedTypes

[lsm5]

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request. If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

The podman team recently tried to build Fedora 33 Beta images for use in our CI and we noticed SSH refused to connect and the daemon logs complained about not finding ssh-rsa in PubKeyAcceptedTypes. And this went away after making the relevant change in /etc/crypto-policies/back-ends/opensshserver.config.

Looks like Fedora 33 has removed ssh-rsa by default, and it's quite likely that other distros will as well, if not already.

It would be nice to not depend on ssh-rsa going forward.

@cevich Please correct me if I got anything wrong ^ :smile:

Potential References

https://github.com/containers/automation_images/pull/26#issuecomment-705780840

[cevich]

Suggestion: Add an ssh_keytype option to allow overriding the temporary key that packer generates. e.g. for the googlecompute builder or globally.

[SwampDragons]

Hi, thanks for reaching out! I agree that it's a good idea to add an option for telling Packer to generate a different key type.

We could probably do this now-ish for Packer with non-rsa type keys, for example ecdsa, and that may actually provide a workaround for users currently bitten by https://github.com/hashicorp/packer/issues/8609 while we wait for the golang team find the time to look at https://github.com/golang/go/issues/39885.

Meanwhile, I think you can work around this yourself by generating your own key and providing it to your builder using the ssh_private_key option

[cevich]

Great, thank for picking this up, and thanks for the suggested work-around. Unfortunately because of the way our specific setup works, it's more difficult to use a custom key. Allowing the less-secure ssh-rsa key was the simpler solution, and security is a relatively low-concern for our images. I doubt that's acceptable for other people though.

[azr]

Hello there, just to say we have binaries that handles this for GCP here.

You can now set the temporary_key_pair_type/temporary_key_pair_bits fields.

I'll leave this issue open until it's fixed for all builders.

[cevich]

Thanks!

[azr]

Hello there, I updated the binaries and the latest ones are here. Seems like to me we are going to keep these. 🙂

[cevich]

FWIW: I just encountered what appears to be the ssh-rsa problem on Ubuntu 20.10 (recently released).

We're not currently setup to install packer from other than the official site, and it seems circle-ci demands an oauth2 token to access the file. Any chance of having this change included in an official alpha/beta release I can point my scripts at?

[azr]

Hey @cevich, #10111 adds that feature for GCP, once it is merged you will be able to use the nightly build the day after it is merged, the PR looks like it's about to be merged so stay tuned ! 🙂

Also the upcoming release should be soon-ish.

[hswong3i]

The key is PubkeyAcceptedKeyTypes, see:

• https://gitlab.com/redhat-crypto/fedora-crypto-policies/-/commit/b298a9e1#49303b123e8d502a8711bccf3ba657fd55abe97c_5_3

[defanator]

Meanwhile, I think you can work around this yourself by generating your own key and providing it to your builder using the ssh_private_key option

Well, this doesn't work for me with packer 1.6.5 and Fedora 33 Cloud Base image (ami-04c4b559df358b45a in eu-central-1). I'm using 2048-bit key with SHA256 encryption, and it allows me to log in EC2 instance launched from the above AMI, but when I'm trying to either use ssh_private_key or put the key to my local ssh agent and enable ssh_agent_auth, I'm getting handshake failures from packer logs with the following log entries at the other side (Fedora 33 instance):

Nov 16 12:43:14 ip-10-1-18-46 sshd[6965]: error: kex_exchange_identification: Connection closed by remote host
Nov 16 12:43:14 ip-10-1-18-46 sshd[6965]: Connection closed by 10.1.2.32 port 43088
Nov 16 12:43:14 ip-10-1-18-46 sshd[6966]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedKeyTypes [preauth]
Nov 16 12:43:14 ip-10-1-18-46 sshd[6966]: Connection closed by authenticating user fedora 10.1.2.32 port 43090 [preauth]

I'm currently looking for at least temporary solution (without modifying original Fedora Cloud Base AMIs) - appreciate any input here.

[azr]

@defanator That PubkeyAcceptedKeyTypes setting needs to be on in client and server side. If that still doesn't work I'd try with other ssh key types, ( packer can generate RSA, DSA, ECDSA and ED25519 temporary keys types for GCP currently ) . There should be at least one that works !

[defanator]

@azr I'm using pre-generated (existing) keypair which works fine on the same client (OS X 10.14) when I'm using system ssh client, but it doesn't work with the one bundled into packer.

We are using original AMI images published by OS vendors to build our custom ones, and currently we can't use Fedora 33 Cloud Base image due to this restriction in packer.

[cevich]

@defanator FWIW, @lsm5 originally discovered the workaround to this problem by switching to an ED25519 key. Just to confirm, are you using the new temporary_key_pair_type option to the googlecompute builder?

[cevich]

Woops...seems you're using AWS not GCP....my bad. I don't see a corresponding option for the amazon-instance builder (but I'm also no familiar with using it).

[defanator]

@cevich yes, I'm using EC2 builder, and the funny fact is that there's no way to import non-RSA key to EC2 via keypair API. But this shouldn't be an issue - as I mentioned previously, my SHA256-encoded key works just fine from anywhere except packer built-in ssh client.

[SwampDragons]

@defanator sounds like you're facing https://github.com/hashicorp/packer/issues/8609. It's an upstream bug and I wrote a patch for the golang crypto library months ago to try to get it fixed and it's waiting for review. You can track it here: https://github.com/golang/go/issues/39885

There's a very messy workaround you can do while we wait: https://github.com/hashicorp/packer/issues/8609#issuecomment-649128385

[defanator]

@SwampDragons appreciate your input here! Yes, it looks more like #8609 but the bad thing is that we can't do the proposed workaround as Fedora 33 official (Cloud Base) images in EC2 are coming with ssh-rsa disabled already. I was thinking of another approach where we would add some bootstrap bits via userdata while launching an instance from AMI and use that for either putting another key for the default user, or even adding another user to let packer log in, but it seems like currently there's no way to specify existing instance ID instead of launching a new one from specified image (AMI). That is, I hope you'll sort this out in #8609 - thanks for your efforts!

[SwampDragons]

Got it. This is starting to affect enough people that we're gonna have to figure something out instead of waiting for Golang to fix it. I'll talk to our security team about options.

[cmollet]

Is there any update on the temporary_key_pair_type for non-GCP builders (Digital Ocean for me specifically)? Setting temporary_key_pair_type = "ed25519" in my digitalocean builder doesn't error out but it seems to have no effect. Running packer -debug, pausing at stepCreateSSHKey and inspecting the resulting key with ssh-keygen -l shows it as an RSA key.

This is a bit of a blocker for those of using Fedora 33. The only workaround I have for now is to run packer in debug mode, pause after the DO Droplet is created, use the DO console to reset the root password, launch a web VNC session, manually tweak the sshd config to add ssh-rsa to PubkeyAcceptedKeyTypes and then resume the build. If anyone has a less tedious approach please let me know!

[cevich]

If anyone has a less tedious approach please let me know!

I dunno about DO, but I imagine it has support for passing in cloud-init metadata? If so, make your edits with boothook or runcmd (docs). For F33, my projects are using a script with something like the following:

    source /etc/os-release
    if [[ "$ID" == "fedora" ]] && [[ "$VERSION_ID" -ge 33 ]]; then
        sed -i -r -e \
            's/^(PubkeyAcceptedKeyTypes )(.*)/\1ssh-rsa,\2/' \
            /etc/crypto-policies/back-ends/opensshserver.config
    fi

[cmollet]

Thanks for the suggestion, @cevich ! Directly modifying the config file didn't work, but running update-crypto-policies --set LEGACY in the user data script did! I did that for the initial Packer build and then restored the crypto-policies back to DEFAULT in the provisioner. FWIW, I still think it's a good idea to allow Packer users to specify key types and bit length for every builder type, but this is a good-enough workaround for me for now.

[chrisroberts]

Hi! We have had this same problem over on the Vagrant side and wanted to share what was learned and how it is being resolved: https://github.com/hashicorp/vagrant/issues/11783#issuecomment-818951984

I had a skim through the golang ssh libs and it looks like the same underlying issue is causing the problems. For RSA keys the signer needs to be updated to default to sha256 and the public key type value used for the building the user auth request needs to be updated to the signature type.

[vectorsigma]

Fedora 33 or 34 images can't be built property in AWS using Packer until this is fixed. It's only a matter of time before other "enterprise" disros upgrade to OpenSSH 8.2+. I'm happy to help test fixes. (I am not letting Packer generate keys, I'm using my own) @SwampDragons thanks for fighting the good fight and keeping momentum going upstream.

[solarsnake]

Just commenting here to say I am encountering this in AWS as well, and would love to the temporary_key_pair_type option to be available for the amazon-ebs builder type.

[SwampDragons]

Last I checked, amazon only supported creating rsa keys through their API, or else I would have just switched the default a year and a half ago :(

[vectorsigma]

If anyone has a less tedious approach please let me know!

I dunno about DO, but I imagine it has support for passing in cloud-init metadata? If so, make your edits with boothook or runcmd (docs). For F33, my projects are using a script with something like the following:

    source /etc/os-release
    if [[ "$ID" == "fedora" ]] && [[ "$VERSION_ID" -ge 33 ]]; then
        sed -i -r -e \
            's/^(PubkeyAcceptedKeyTypes )(.*)/\1ssh-rsa,\2/' \
            /etc/crypto-policies/back-ends/opensshserver.config
    fi

I'm doing something similar now, but instead of seding files, there's a small utility at least in Fedora 34, that will do this properly with one command: It's called update-crypto-policies.

For example, in the ebs builder, I pass a script path to user_data_file that contains nothing more than:

#!/bin/bash
sudo update-crypto-policies --set LEGACY

That gets Packer back into the instance to provision the image. Once that's done, and you can revert to using the real openssh client, you can choose to switch back as follows:

#!/bin/bash
sudo update-crypto-policies --set DEFAULT

[lorengordon]

@SwampDragons AWS now supports ec25519 keypairs: https://aws.amazon.com/about-aws/whats-new/2021/08/amazon-ec2-customers-ed25519-keys-authentication/

[danbarr]

@lorengordon sadly, ed25519 is not FIPS-validated, so still not usable on FIPS-enabled systems

[lorengordon]

@danbarr Oh, wow. Thanks for that. Ok. And now I no longer care about ed25519 support. I'll have to disable FIPS in the image, or maybe locally-generate an ecdsa key and use ssh_private_key_file with userdata to setup the public key. Bummed.

[danbarr]

@lorengordon Indeed; I saw that announcement, got all excited, and then immediately sad again. Needed them to also do ecdsa-sha2-nistp256 or higher.

RSA-2048 keys do still work in general, as long as your SSH client is using sha256 signatures - any recent OpenSSH does, and PuTTY does as of 0.75. The issue is that the Go library used by Packer (and others) does not.

[lorengordon]

Aha, ok. And my putty version is out-of-date. So at least that explains the other part of my problem, where I couldn't get even my rsa key to work...

[vectorsigma]

FWIW, now that both Fedora 33+ and AWS support ED25519, in my Packer scripts, I set temporary_key_pair_type to ed25519 and all is well again with no bothersome user_data hacks dealing with crypto policies.

[ProFfeSsoRr]

@vectorsigma what type of builder you use? amazon-ebs ignore temporary_key_pair_type option and always created rsa.

[crabby345]

@vectorsigma what type of builder you use? amazon-ebs ignore temporary_key_pair_type option and always created rsa.

I found the same. RSA created every time. I created my own ed25519 pair and then just supplied packer the key to get around it.

[jvperrin]

I was experiencing this same problem, but using the recently released Packer 1.8.0 that includes a newer version of hashicorp/packer-plugin-amazon (including https://github.com/hashicorp/packer-plugin-amazon/pull/179) and specifying "temporary_key_pair_type": "ed25519" with the amazon-ebs builder has worked great to fix this!

[Miosame]

is there any tracked issue for the other builders to implement pair type besides aws?

[puetzk]

Add ubuntu 22.04 LTS "Jammy Jellyfish" to the distributions that packer's current ssh client can no longer communicate with (at least not with ssh-rsa keys)

[nywilken]

Add ubuntu 22.04 LTS "Jammy Jellyfish" to the distributions that packer's current ssh client can no longer communicate with (at least not with ssh-rsa keys)

Thanks for the ping here @puetzk. Fixes to the Amazon and Azure plugins were released yesterday. You can use any of the respective releases to resolve the authentication issue.

• https://github.com/hashicorp/packer-plugin-amazon/releases/tag/v1.1.0
• https://github.com/hashicorp/packer-plugin-azure/releases/tag/v1.0.7

If there is a specific packer-plugin-* that you would like to see that patch applied to please drop a list here so that we can work to prioritize their release with the patch.

[nywilken]

is there any tracked issue for the other builders to implement pair type besides aws?

We don't have an master issue for tracking this as the decision to support has been mostly left to the builder plugins to support. The configuration value is already available to all builders they just need to use it when generating their respective ssh keys. That said if you have a list of builders that you would like to see this implemented in please feel free to open an issue with the list so we can work on prioritizing. Cheers!

[nywilken]

Hi folks a potential fix for this issue has been merged into Packer main. It is scheduled to be released next week but will be available in the next nightly release. The SSH communication is handled by the Packer Plugin SDK within each of the builders. So even though Packer has the latest SDK the plugin responsible for executing the builder will need to be updated as well in order for the fix to work. We are tracking individual plugin updates in #11761.

To download the latest plugin you can run packer init against a HCL file that is using the required_plugins{...} block with the updated plugin version. Or you can run the packer plugins install command to get the latest version for a given plugin. For example you can run packer plugins install github.com/hashicorp/docker to install the latest version of Docker.

note: The latest nightly has the fix https://github.com/hashicorp/packer/pull/11712 which should help folks using JSON

[nywilken]

Oh one more thing 😄

If you are still running into issues with the proposed fix please feel free to drop a comment on the thread and we will gladly reopen. Thanks again to everyone for providing workarounds and for your patience.

[m62534]

Hello, is there a fix for this for vmware-iso? Running into issues in Ubuntu 22.04 (Running vmware-iso version 1.0.7)

[tomsa95]

Hello, is there a fix for this for vmware-iso? Running into issues in Ubuntu 22.04 (Running vmware-iso version 1.0.7)

Hello, the same for me with vmware-iso, packer version: 1.8.4

[nickvth]

A Remote Server Workaround If you can't change anything on the local computer, or don't want to use a new key, and want to re-enable RSA on the local computer, edit the file /etc/ssh/sshd_config on the remote computer and add this line:

HostKeyAlgorithms +ssh-rsa PubkeyAcceptedKeyTypes +ssh-rsa This will allow the use of unsafe RSA key you already have.

[inflatador]

I'm still running into this issue using the qemu builder:

SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

Would it be possible to address this on all builders? I see @tomsa95 's comment above that it still affects the vmware builder.

[rubenanapu]

Using Packer 1.8.0 allowed me to use the temporary_key_pair_type key, and I was able to create the AMI (Amazon Cloud).

I was experiencing this same problem, but using the recently released Packer 1.8.0 that includes a newer version of hashicorp/packer-plugin-amazon (including hashicorp/packer-plugin-amazon#179) and specifying "temporary_key_pair_type": "ed25519" with the amazon-ebs builder has worked great to fix this!

I tried before with Packer 1.2.4, 1.3.2, 1.11.1 and none of them allowed me to use the temporary_key_pair_type key, but with Packer 1.8.0 worked.

Thank you very much, @jvperrin.