Accessing distant VirtualBox through SSH fails with recent OpenSSH servers

[bgandon]

When the host CPI property is set (along with username and private_key), the CPI acts on a distant VitrualBox installation connecting with SSH.

This features doesn't work out-of-the-box anymore. Indeed the vbox CPI v0.4.2 uses a deprecated ssh-rsa singing algorithm when authenticating to the distant SSH server. Whereas recent OpenSSH installations are setup to refuse that signing algorithm by default, which leads to connection failures.

Example of failure

Here is an example with a bosh upload-stemcell operation made against a Bosh Director, with properly-configured VirtualBox CPI (named parent-host-vbox) that tries to connect to the “example-vbox-host” distant VirtualBox host:

$ bosh upload-stemcell --sha1 "7724ce4272dd8f19b44584a17d31595eac7595e5"   "https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=621.125"
...
...
Task 982 | 15:19:56 | Update stemcell: Checking if this stemcell already exists (cpi: parent-host-vbox) (00:00:00)
Task 982 | 15:19:56 | Update stemcell: Uploading stemcell bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125 to the cloud (cpi: parent-host-vbox) (00:00:11)
                    L Error: CPI error 'Bosh::Clouds::CloudError' with message 'Importing stemcell from '/var/vcap/data/director/tmp/stemcell20240211-11466-yi90el/image': Creating stemcell parent: Connecting via SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain' in 'create_stemcell' CPI method (CPI request ID: 'cpi-307703')
Task 982 | 15:20:07 | Error: CPI error 'Bosh::Clouds::CloudError' with message 'Importing stemcell from '/var/vcap/data/director/tmp/stemcell20240211-11466-yi90el/image': Creating stemcell parent: Connecting via SSH: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain' in 'create_stemcell' CPI method (CPI request ID: 'cpi-307703')

Task 982 Started  Sun Feb 11 15:19:30 UTC 2024
Task 982 Finished Sun Feb 11 15:20:07 UTC 2024
Task 982 Duration 00:00:37
Task 982 error

Uploading remote stemcell 'https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=621.125':
  Expected task '982' to succeed but state is 'error'

Exit code 1

It fails and on the “example-vbox-host” the logs show that the ssh-rsa key algorithm is refused:

benjamin@example-vbox-host:~$ sudo tail -F -n0 /var/log/auth.log
Feb 11 15:20:07 example-vbox-host sshd[356145]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Feb 11 15:20:07 example-vbox-host sshd[356145]: Connection closed by authenticating user benjamin <REDACTED-DIRECTOR-IP> port 60962 [preauth]

CPI SSH settings are fine, though

When connecting with OpenSSH from the Bosh Director, using the CPI settings, it works. But the signing algorithm is more recent: rsa-sha2-512.

bosh/0:~# apt update -qq && apt install -y -qq jq
bosh/0:~# jq -r .PrivateKey /var/vcap/jobs/virtualbox_cpi/config/cpi.json > ssh-key-from-config
bosh/0:~# chmod 600 ssh-key-from-config 
bosh/0:~# ssh -vvvv -i ssh-key-from-config $(jq -r .Username /var/vcap/jobs/virtualbox_cpi/config/cpi.json)@$(jq -r .Host /var/vcap/jobs/virtualbox_cpi/config/cpi.json) cat /etc/hostname
...
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:PP64/8ntZnYRHjjBFkIx8QNf2roVy0YrApSPLCPwNi8
...
example-vbox-host
...

Workaround relaxing the ssh config

This StackExchange answer is a great source of understanding. Based on the suggested solution, we can modify the sshd config on the vbox host to accept the old signing algorithm:

# cat > /etc/ssh/sshd_config.d/accept-ssh-rsa-algorithm.conf
PubkeyAcceptedAlgorithms +ssh-rsa
# systemctl restart sshd
# systemctl status sshd

Then the same bosh upload-stemcell operation just works fine:

$ bosh upload-stemcell --sha1 "7724ce4272dd8f19b44584a17d31595eac7595e5"   "https://bosh.io/d/stemcells/bosh-vsphere-esxi-ubuntu-xenial-go_agent?v=621.125"
...
...
Task 1112 | 14:42:21 | Update stemcell: Uploading stemcell bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125 to the cloud (cpi: parent-host-vbox) (00:00:25)
Task 1112 | 14:42:46 | Update stemcell: Save stemcell bosh-vsphere-esxi-ubuntu-xenial-go_agent/621.125 (sc-d7340402-90bb-44dd-4ea0-dfd8dfd35472) (cpi: parent-host-vbox) (00:00:00)

Task 1112 Started  Tue Feb 13 14:42:01 UTC 2024
Task 1112 Finished Tue Feb 13 14:42:46 UTC 2024
Task 1112 Duration 00:00:45
Task 1112 done

Succeeded

Conclusion

The VitualBox CPI v0.4.2 is compiled using Golang v1.17.2, which old SSH libraries are still using by default a deprecated ssh-rsa singing algorithm when authenticating to the distant VirtualBox host.

Recent OpenSSH server installation expect more secure rsa-sha2-256 or rsa-sha2-512 algorithms instead. These are certainly the default in recent Golang SSH libraries.

In order for the SSH feature to work again out-of-the-box, we need the VirtualBox CPI to be recompiled using a recent Go version.

[bgandon]

The Gstack-released VirtualBox CPI v0.4.3 fixes this issue: https://github.com/gstackio/bosh-virtualbox-cpi-release/releases/tag/v0.4.3

[bgandon]

This issue will be fixed by PR #35.