search

crc-org / crc

CRC is a tool to help you run containers. It manages a local OpenShift 4.x cluster, Microshift or a Podman VM optimized for testing and development purposes
https://crc.dev
Apache License 2.0
1.25k stars 237 forks source link

Incompatibility between the new SSH cipher policies in FCOS 33 and golang.org/x/crypto/ssh #1747

Closed cgruver closed 3 years ago

cgruver commented 3 years ago

It looks like the id_rsa file is not getting created in the .crc/machine directory.

General information

CRC version

CodeReady Containers version: 1.19.0-OKD+8d69386
OpenShift version: 4.6.0-0.okd-2020-11-27-200126 (embedded in executable)

Steps to reproduce

  1. The bundle was built with:

    OKD_VERSION=4.6.0-0.okd-2020-11-27-200126
OPENSHIFT_PULL_SECRET_PATH=/tmp/pull_secret.json

    The SNC revision is from my fork: https://github.com/cgruver/snc The revision is: 4.1.14-157-g291ef43

    The single node cluster built successfully, and was usable. The bundle built successfully.

  2. CRC was built with:

    OKD_BUILD=true
BUNDLE_VERSION=4.6.0-0.okd-2020-11-27-200126
BUNDLE_DIR=/crc-build/snc
make embed_bundle

    The CRC build completed successfully.

  3. The setup completes with no errors: Tested on MacOS 11 and CentOS 8

    crc setup

    It unpacks the bundle and prepares the environment.

  4. The cluster start fails with ssh errors:

INFO Starting CodeReady Containers VM for OpenShift 4.6.0-0.okd-2020-11-27-200126... 
DEBU Updating CRC VM configuration                
(crc) Calling .GetConfigRaw
(crc) Calling .Start
(crc) DBG | Starting VM crc
(crc) DBG | Validating network
(crc) DBG | Validating storage pool
(crc) DBG | GetIP called for crc
(crc) DBG | Getting current state...
(crc) DBG | IP address: 192.168.130.11
(crc) Found IP for machine: 192.168.130.11
(crc) Calling .GetState
(crc) DBG | Getting current state...
(crc) Calling .GetIP
(crc) DBG | GetIP called for crc
(crc) DBG | Getting current state...
(crc) DBG | IP address: 192.168.130.11
DEBU Waiting until ssh is available               
DEBU retry loop: attempt 0                        
DEBU About to run SSH command:
exit 0             
Cannot read private ssh key /home/crc-user/.crc/machines/crc/id_rsa
DEBU SSH command results: err: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain, output:  
DEBU error: Temporary error: ssh command error:
command : exit 0
err     : ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
output  :  - sleeping 1s 
DEBU retry loop: attempt 1                        
DEBU About to run SSH command:
exit 0             
Cannot read private ssh key /home/crc-user/.crc/machines/crc/id_rsa
DEBU SSH command results: err: ssh: handshake failed: ssh: unable to...
etc...  Until it times out.
gbraad commented 3 years ago

Starting CodeReady Containers VM for OpenShift 4.6.0-0.okd-2020-11-27-200126

Hmmmm... this is not the correct issue tracker for OKD-related issues ;-)

It looks like the id_rsa file is not getting created in the .crc/machine directory.

Did you perform a crc delete -f beforehand? The key should be dropped in the ~/.crc/machines/crc folder, but perhaps it is confused by an existing (partial) VM?

cgruver commented 3 years ago

Yeah, this was a completely clean build. The ~/.crc/machines/crc/id_rsa file is not getting created, but I don't see any errors that indicate why...

I ran crc setup and crc start with debug log level to look for clues.

guillaumerose commented 3 years ago

If this is the first run, the error is "normal" as we are using 2 possible keys to connect. The other one is in the extracted bundle.

If this is a second run, it was aborted before or during the new key generation I suppose.

cgruver commented 3 years ago

This issue turns out to be an incompatibility between the new SSH cipher policies in FCOS 33 and golang.org/x/crypto/ssh

Reverting the policies in FCOS 33 to support RSA-SHA1 fixes the issue.

cat > /etc/ssh/sshd_config.d/10-fcos-insecure-rsa-key.conf <<EOF
> # For now allow RSA-SHA1 keys.
> # https://github.com/coreos/coreos-assembler/issues/1772
> PubkeyAcceptedKeyTypes=+ssh-rsa
> EOF
guillaumerose commented 3 years ago

wow! nice catch.

cfergeau commented 3 years ago

Keeping this open, we'll need to fix it somehow edit: maybe all that is needed is to switch https://github.com/code-ready/crc/blob/master/pkg/crc/ssh/keys.go#L29 to https://golang.org/pkg/crypto/ed25519/#GenerateKey and to also do a similar change in snc

praveenkumar commented 3 years ago

@cfergeau does this change also going to work with RHCOS ?

cfergeau commented 3 years ago

This will need to be tested, but I expect RHEL8 to support ecdsa and ed25519 keys in addition to rsa keys. man ssh-keygen on a rhel 8.3 system has ssh-keygen [-q] [-b bits] [-t dsa | ecdsa | ed25519 | rsa], openssh RHEL builds from 2013/2014 have mentions of ecdsa/ed25519, so I'm quite confident it will be all fine.

cgruver commented 3 years ago

I fixed it!!!

I switched snc and crc to use ecdsa keys, and it appears to work.

I'll submit a PR in the morning so you guys can vet it out.

cgruver commented 3 years ago

Fixed by: https://github.com/code-ready/snc/pull/289, https://github.com/code-ready/crc/pull/1757