ssh handshake failure after changing system-wide crypto policy to FIPS

[pfuntner]

Overview of the Issue

After running an Ansible playbook that performs update-crypto-policies --set FIPS and reboots an RHEL8 AWS instance, packer cannot reconnect to the instance.

Reproduction Steps

1. git clone https://gist.github.com/19d6e0abbea44001c5d1c4e409e29dd8.git
2. cd 19d6e0abbea44001c5d1c4e409e29dd8
3. Update ssh_keypair_name and ssh_private_key_file in aws-ebs-ansible.json for an existing AWS key pair. The issue will manifest if packer creates a temporary ssh keypair but if you use an existing pair, you can ssh to the directly instance even when packer cannot.
4. packer build -var-file vars-rhel8.json aws-ebs-ansible.json
5. Observe that packer cannot reconnect to after the system-wide policy is updated and the instance is rebooted (see debug.log for complete log):

   .
   .
   .
   amazon-ebs: changed: [default]
   amazon-ebs:
   amazon-ebs: TASK [Wait for reboot to complete] *********************************************
   amazon-ebs: ok: [default -> localhost]
   amazon-ebs:
   amazon-ebs: TASK [Get uptime] **************************************************************
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: SSH proxy: accepted connection
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: authentication attempt from 127.0.0.1:42116 to 127.0.0.1:37060 as ec2-user using none
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: authentication attempt from 127.0.0.1:42116 to 127.0.0.1:37060 as ec2-user using publickey
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: new env request: LANG=en_US.UTF-8
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: new exec request: /bin/sh -c 'echo ~ec2-user && sleep 0'
   2020/01/15 07:11:10 packer-builder-amazon-ebs plugin: [DEBUG] Opening new ssh session
   2020/01/15 07:11:10 packer-builder-amazon-ebs plugin: [ERROR] ssh session open error: 'EOF', attempting reconnect
   2020/01/15 07:11:10 packer-builder-amazon-ebs plugin: [DEBUG] reconnecting to TCP connection for SSH
   2020/01/15 07:11:10 packer-builder-amazon-ebs plugin: [DEBUG] handshaking with SSH
   2020/01/15 07:11:10 [INFO] 0 bytes written for 'stdout'
   2020/01/15 07:11:10 [INFO] 0 bytes written for 'stderr'
   2020/01/15 07:11:10 [INFO] 0 bytes written for 'stdin'
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: [INFO] 0 bytes written for 'stdout'
   2020/01/15 07:11:10 packer-provisioner-ansible plugin: [INFO] 0 bytes written for 'stderr'
   ==> amazon-ebs: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
   .
   .
   .

6. packer hangs at this point but I can use ssh to the AWS instance directly (ec2-user@54.209.97.87) using my existing private key.
7. Use Ctrl-C to bring stop packer and clean up the AWS temporary instance

Packer version

Packer v1.5.1

Simplified Packer Buildfile

file-aws-ebs-ansible.json

Operating system and Environment details

Driving system is CentOS 7 (3.10.0-957.27.2.el7.x86_64)

Log Fragments and crash.log files

debug.log

[ODINServ]

I have the same issue with a normal RHEL8.1 machine with FIPS enabled.

I've downloaded packer 1.5.4 from the website and we are running packer locally, not AWS. I suspect that this issue is not isolated to the aws builder.

[dbilling]

The best I can tell, this issue is because Packer appears to be using golang's ssh which is signing with SHA-1 during the ssh handshake (i.e. using signature type ssh-rsa instead of rsa-sha2-256 or rsa-sha2-512). Such signatures are prohibited when system-wide crypto policy is set to FIPS on a RHEL 8 or CentOS 8 system, thus the connect error.

Furthermore, it appears that OpenSSH 8.2 (released 13 Feb 2020) is completely removing support for signature type ssh-rsa, so this is going to become another area of incompatibility... that is, packer will not be able to connect to any OpenSSH 8.2 based system. see https://lwn.net/Articles/812537/ for more OpenSSH 8.2 info.

Golang ssh appears to recently have recently started looking into this, as seen here: https://github.com/golang/go/issues/37278 This issue discusses a potential workaround for those using golang ssh, that perhaps packer could consider implementing: https://github.com/golang/go/issues/36261

Note that rsa keys are still allowed by both OpenSSH 8.2 and RHEL with systemwide crypto policy of FIPS. It's the hash function that is the problem. This was explained in the OpenSSH 8.2 forums: "authenticating yourself using an RSA key involves signing a message with that key to prove that you own it. RSA signatures normally work by first hashing the plaintext message with some hash function and then applying the RSA algorithm to the result. The hash function can be basically anything reasonable, but it needs to be agreed by both parties. From this you can see that there doesn't need to be any particular link between hash function and key size."

For now, since Packer will currently only sign with hash SHA-1, packer will remain unable to connect to systems with either >= OpenSSH 8.2 and/or RHEL8/CentOS8 with system-wide crypto policy FIPS or FUTURE, as they prohibit this hash in their accepted signature types.

This issue really needs some attention ASAP...

[lnbnatera]

This issue isn't isolated to aws builder. I experienced the same with vsphere-iso and virtualbox-iso when I attempted to build CentOS8 and RedHat8 images with FIPS enabled from kickstart.

2020/04/18 08:48:20 packer.exe plugin: [INFO] Attempting SSH connection to <my_IP>:<my_Port>...
2020/04/18 08:48:20 packer.exe plugin: [DEBUG] reconnecting to TCP connection for SSH
2020/04/18 08:48:20 packer.exe plugin: [DEBUG] handshaking with SSH
2020/04/18 08:48:21 packer.exe plugin: [DEBUG] SSH handshake err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain

While build is in progress, attempting an ssh connection from an CentOS7 and RedHat7 with FIPS disabled still works. Any version lower than those such as CentOS6 or RedHat6 fails to connect. As circumstantial as it seems, this scenario further points the culprit being the ssh communicator of packer. And as stated earlier, it is safe to say that packer will be unable to connect to any FIPS-enabled OS until the internal packer ssh communicator is fixed.

[ghunsl]

I'm not sure this will help but, I had a similar issue after enabling the FIPS on RHEL 7. The packer could not reconnect back to the instance. If I skip the reboot step after enabling the FIPS, the packer was able to continue to create the image; however, I was not able to ssh back to the instance once the image was created and I started the instance from created image.

It turned out that sshd daemon failed to start properly due to unrecognized configs for MACs in /etc/ssh/sshd_config file. Initially, I had MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com and I changed this to MACs hmac-sha2-512,hmac-sha2-256 and sshd daemon was able to start properly. I changed to those values based on the document from https://access.redhat.com/solutions/2208461 which states FIPS approved Ciphers and MACs. Also, some helpful link is https://www.ssh.com/manuals/server-admin/44/Ciphers_and_MACs.html

I hope this helps a little.

[dbilling]

I'm not sure this will help but, I had a similar issue after enabling the FIPS on RHEL 7. The packer could not reconnect back to the instance. If I skip the reboot step after enabling the FIPS, the packer was able to continue to create the image; however, I was not able to ssh back to the instance once the image was created and I started the instance from created image.

Hi Ghunsl - Great to hear you got this working on RHEL7. However, systemwide crypto policy was shipped in CentOS 8/RHEL 8, so I think we are hitting a different issue. Specifically, we are talking about packer not being able to connect to machines where the SSH setting for PubkeyAcceptedKeyTypes does not include "ssh-rsa". "ssh-rsa" is prohibited by FIPS because it uses SHA-1.

[SwampDragons]

Thanks for opening this folks, and sorry it took so long to give it attention. Just to make sure I fully understand the cases for reproduction -- are you all using your own private keys made with a non- "ssh-rsa" algorithm, or have those of you who've hit this issue been letting Packer generate the keys for you?

[dbilling]

Thanks for opening this folks, and sorry it took so long to give it attention. Just to make sure I fully understand the cases for reproduction -- are you all using your own private keys made with a non- "ssh-rsa" algorithm, or have those of you who've hit this issue been letting Packer generate the keys for you?

We are using packer to generate RSA keys for us. Note that PubkeyAcceptedKeyTypes specifies both the key type and hash type to use for the public key verification algorithm. "ssh-rsa" uses rsa keys with the SHA-1 hash algorithm. The OS system we are connecting packer to has deprecated the use of ssh-rsa because it is now (or will soon be) insecure due to the SHA-1 hash. If packer supported a more modern ssh signature hash algorithm such as rsa-sha2-256 or rsa-sha2-512, the connection would work with the exact same RSA keys. But packer doesn't support any of the newer secure hash algorithms for this purpose... that's what we hope to get fixed...

[SwampDragons]

Update: I'm trying to adapt the suggested AlgorithmSigner wrapping method to something that'll work inside of Packer.

In the meantime, I have a workaround that will unblock y'all, at the cost of some extra build/pipeline complexity.

What you need to do is divide your build into two stages (two discrete Packer "builds"); in the first stage, you enable FIPS and then shut down the machine. BUT before you do, upload a new non-RSA key to your instance.

e.g. Before your build create an ecdsa key:

ssh-keygen -f ./example-key-ecdsa -t ecdsa -b 256

Then add a file provisioner to upload that key:

    {
      "type": "file",
      "destination":"/home/{{user `ssh_username`}}/.ssh/authorized_keys",
      "source": "/home/username/.ssh/example-key-ecdsa.pub"
    },

Then call your provisioner that updates the crypto policy that disables sha1 as your FINAL provisioning step. (in the linked gist, that would mean modifying harden.yml to exit after the zzz step rather than waiting for a restart, and removing the provisioner after it that cleans up authorized keys.)

When that build completes, you'll get an output AMI, but it won't be your "final" ami.

Now you can launch a new build using that output-AMI as your source_ami, and provide your special key as the connection key via ssh_private_key_file.

example:

{
    "builders": [
        {
            "ami_name": "packer-example-stage-2",
            "force_deregister": true,
            "instance_type": "t2.micro",
            "source_ami": " {{user `output_from_stage_1` }}",          
            "ssh_username": "ec2-user",
            "ssh_private_key_file": "~/.ssh/example-key-ecdsa",
            "type": "amazon-ebs"
        }
    ],
  "provisioners": [
    {
        "type": "shell-local",
        "inline": ["echo it is not pretty but it works"]
    },
  ]
}

I know it's a mess of a workaround, but hopefully either I'll be able to figure out a solution on the Packer end or the x/crypto library will merge their fix.

[SwampDragons]

I've got some good news and bad news. The bad news is that I can't fix this inside of Packer without re-implementing a huge portion of the golang crypto/ssh library, which I don't want to do. The good news is that I've opened a patch to try to fix this upstream: https://go-review.googlesource.com/c/crypto/+/240717

I'm not sure how long it will take to get merged. In the meantime, I'll talk with our security team about the implications of maintaining our own fork of the crypto library in order to un-block the Packer users who are affected by this.

[SwampDragons]

Hi everyone, a brief update on this: the HashiCorp security team is wary of Packer having to maintain a fork of a core crypto library, so for now we're going to watch and wait and see what happens with the PR. That said, I know some of you are blocked on this; if y'all want, I can create a developmental Packer build with the crypto patch applied so you don't have to go through the tedious workaround above. Just let me know if there's interest in that, and I'll work on it now-ish instead of when the golang crypto team gets around to reviewing my changelist.

[SwampDragons]

I still haven't seen movement on my linked patch, but I'll be sure to update this issue when I hear anything.

[Promaethius]

@SwampDragons there may be a more elegant solution than maintaining your own crypto right now: https://hub.docker.com/r/goboring/golang

[jeremymcgee73]

Any updates on this? Just curious if there are any new workarounds. @SwampDragons

[SwampDragons]

Unfortunately I'm still waiting on a review over at https://github.com/golang/go/issues/39885; I haven't spent much time trying to build another workaround into Packer because I haven't seen a whole lot of action on this ticket, but we may need to try to figure something else out if it's starting to affect more people.

[jeremymcgee73]

Thanks for the response! Woof. That is annoying :-)

[danbarr]

Gave my +1 over on related issue #10074, but I'll throw another vote here too. We're required to enable FIPS mode on our systems and since AWS still doesn't support ECDSA/Ed25519 keypairs with EC2, RSA with SHA2-256 or SHA2-512 is our only option. I've had to do some nasty kludges to keep our Packer builds working through our provisioning steps.

[lorengordon]

I believe the golang crypto library has updated the client-side to support a compatible algorithm now. If packer updates to the current version of the library, perhaps this old ticket could be closed?

[feiyushi]

I believe the golang crypto library has updated the client-side to support a compatible algorithm now. If packer updates to the current version of the library, perhaps this old ticket could be closed?

According to this reply, someone already tested it: https://github.com/golang/go/issues/39885#issuecomment-1067467084 Can packer also integrate the change? One project mentioned in the issue which had a workaround also integrated the changes and removed the workaround: https://github.com/go-gitea/gitea/pull/19097

[nywilken]

Thanks for all the help in communicating various workarounds. The latest Packer SDK has been patched with the Golang crypto/ssh fix . We will be rolling out the fixes to each individual plugin as the changes get merged.

Once all plugins have been updated we I’ll work to get a release of Packer core out with the update crypto fix. For those using HCL pinning each plugin to the latest available release should be enough to get the fix in place.

[nywilken]

Hi folks a potential fix for this issue has been merged into Packer main. It is scheduled to be released next week but will be available in the next nightly release. The SSH communication is handled by the Packer Plugin SDK within each of the builders. So even though Packer has the latest SDK the plugin responsible for executing the builder will need to be updated as well in order for the fix to work. We are tracking individual plugin updates in #11761.

To download the latest plugin you can run packer init against a HCL file that is using the required_plugins{...} block with the updated plugin version. Or you can run the packer plugins install command to get the latest version for a given plugin. For example you can run packer plugins install github.com/hashicorp/docker to install the latest version of Docker.

note: The latest nightly has the fix https://github.com/hashicorp/packer/pull/11712 which should help folks using JSON or HCL without a required plugins block.

[nywilken]

Oh one more thing 😄

If you are still running into issues with the proposed fix please feel free to drop a comment on the thread and we will gladly reopen. Thanks again to everyone for providing workarounds and for your patience.