SSH hardening, removing outdated ciphers, kex algorithms, etc

[immanuelfodor]

• I have read the contributing guide lines at https://github.com/opnsense/core/blob/master/COoNTRIBUTING.md
• I have searched the existing issues and I'm convinced that mine is new.

Hi Everyone,

There is a forum post from somebody else recommending opening an issue for ssh hardening but there are no issues to be found after searching, so here is mine.

I'd like to harden my ssh connection on the OPNsense server side, how can I do that, which config files to edit to persist the change?

Would like to implement the recommendations provided by this tool: https://github.com/jtesta/ssh-audit.git

[AdSchellevis]

The service responsible for writing the configuration file is https://github.com/opnsense/core/blob/e4393e0decb62ea9a02d88ea3af1963267235d06/src/etc/inc/plugins.inc.d/openssh.inc#L72

Unfortunately this is still legacy code and doesn't support overlays, but if you can collect a list of options that should be changed in your opinion together with the current default and desired options, we can take a look if we can turn this ticket in a feature request and schedule it.

We would like to avoid breaking existing setups, so most options likely need to be selectable from here https://github.com/opnsense/core/blob/e4393e0decb62ea9a02d88ea3af1963267235d06/src/www/system_advanced_admin.php#L638

[immanuelfodor]

Generally, I'd like to have access to these three properties of the sshd_config file:

KexAlgorithms
Ciphers
MACs

For me, the desired state would be to set these to for something like this, copying from a fairly hardened config:

KexAlgorithms curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com

The above still has two actionable recommendations in the ssh-audit tool's report but it's much much better than the current default. It would also definitely break older ssh clients, this is why I'd let the user to edit these lines on their own as part of an advanced configuration block.

[AdSchellevis]

we don't support advanced configuration blocks, unless the service supports include directories and the user can drop its modifications in a file on disk. Free text fields are impossible to validate and sooner or later turn into various security issues impossible to fix.

If we can come up with a list of options to add, we might consider adding them. The above list certainly helps in that regard.

[immanuelfodor]

Fair enough, I can completely see your point why a text input is not a good solution on the long run.

Then I can imagine a checkbox field where the user can select their chosen kex, cipher and mac algorithms from a predefined lists what the currently used SSH server supports. There could be a "Reset to default" button to load the current OPNsense defaults that are considered to be the most compatible options. Upon clicking the button, it puts or removes the ticks to all the checkboxes in the field according to the defaults it ships with. The user can then modify the selection according to their liking, and this is used on save to generate these three options in the SSH server config.

Getting what kex, cipher and macs the currently installed SSH supports: https://superuser.com/questions/868998/how-can-i-find-a-list-of-macs-ciphers-and-kexalgorithms-that-my-openssh-client/869005#869005 These lists could provide the values to the checkbox field.

Furthermore, the audit tool is just one Python file, maybe it's possible to integrate it to self-test the configuration. Like the heath test under the firmware menu to give feedback on the settings currently set up. Running like ./ssh-audit.py localhost $SSH_PORT or ./ssh-audit.py --verbose $ANY_IP_OR_HOST $ANY_SSH_PORT inside OPNsense and grabbing its output.

For the current OPNsense release, the tool says the following:

# general
(gen) banner: SSH-2.0-OpenSSH_8.1 FreeBSD-openssh-portable-8.1.p1,1
(gen) software: OpenSSH 8.1 running on FreeBSD
(gen) compatibility: OpenSSH 7.4+, Dropbear SSH 2018.76+
(gen) compression: enabled (zlib@openssh.com)

# key exchange algorithms
(kex) curve25519-sha256                     -- [info] available since OpenSSH 7.4, Dropbear SSH 2018.76
(kex) curve25519-sha256@libssh.org          -- [info] available since OpenSSH 6.5, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp256                    -- [fail] using weak elliptic curves
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp384                    -- [fail] using weak elliptic curves
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) ecdh-sha2-nistp521                    -- [fail] using weak elliptic curves
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(kex) diffie-hellman-group-exchange-sha256 (2048-bit) -- [info] available since OpenSSH 4.4
(kex) diffie-hellman-group16-sha512         -- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) diffie-hellman-group18-sha512         -- [info] available since OpenSSH 7.3
(kex) diffie-hellman-group14-sha256         -- [info] available since OpenSSH 7.3, Dropbear SSH 2016.73
(kex) diffie-hellman-group14-sha1           -- [warn] using weak hashing algorithm
                                            `- [info] available since OpenSSH 3.9, Dropbear SSH 0.53

# host-key algorithms
(key) rsa-sha2-512 (2048-bit)               -- [info] available since OpenSSH 7.2
(key) rsa-sha2-256 (2048-bit)               -- [info] available since OpenSSH 7.2
(key) ssh-rsa (2048-bit)                    -- [fail] using weak hashing algorithm
                                            `- [info] available since OpenSSH 2.5.0, Dropbear SSH 0.28
(key) ecdsa-sha2-nistp256                   -- [fail] using weak elliptic curves
                                            `- [warn] using weak random number generator could reveal the key
                                            `- [info] available since OpenSSH 5.7, Dropbear SSH 2013.62
(key) ssh-ed25519                           -- [info] available since OpenSSH 6.5

# encryption algorithms (ciphers)
(enc) chacha20-poly1305@openssh.com         -- [info] available since OpenSSH 6.5
                                            `- [info] default cipher since OpenSSH 6.9.
(enc) aes128-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes192-ctr                            -- [info] available since OpenSSH 3.7
(enc) aes256-ctr                            -- [info] available since OpenSSH 3.7, Dropbear SSH 0.52
(enc) aes128-gcm@openssh.com                -- [info] available since OpenSSH 6.2
(enc) aes256-gcm@openssh.com                -- [info] available since OpenSSH 6.2

# message authentication code algorithms
(mac) umac-64-etm@openssh.com               -- [warn] using small 64-bit tag size
                                            `- [info] available since OpenSSH 6.2
(mac) umac-128-etm@openssh.com              -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-256-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha2-512-etm@openssh.com         -- [info] available since OpenSSH 6.2
(mac) hmac-sha1-etm@openssh.com             -- [warn] using weak hashing algorithm
                                            `- [info] available since OpenSSH 6.2
(mac) umac-64@openssh.com                   -- [warn] using encrypt-and-MAC mode
                                            `- [warn] using small 64-bit tag size
                                            `- [info] available since OpenSSH 4.7
(mac) umac-128@openssh.com                  -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 6.2
(mac) hmac-sha2-256                         -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 5.9, Dropbear SSH 2013.56
(mac) hmac-sha2-512                         -- [warn] using encrypt-and-MAC mode
                                            `- [info] available since OpenSSH 5.9, Dropbear SSH 2013.56
(mac) hmac-sha1                             -- [warn] using encrypt-and-MAC mode
                                            `- [warn] using weak hashing algorithm
                                            `- [info] available since OpenSSH 2.1.0, Dropbear SSH 0.28

# fingerprints
(fin) ssh-ed25519: XXXXXXXXXXXXXXXXXXXXXXX
(fin) ssh-rsa: SHA256: XXXXXXXXXXXXXXXXXXXXXXX

# algorithm recommendations (for OpenSSH 8.1)
(rec) -ecdh-sha2-nistp256                   -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp384                   -- kex algorithm to remove 
(rec) -ecdh-sha2-nistp521                   -- kex algorithm to remove 
(rec) -ecdsa-sha2-nistp256                  -- key algorithm to remove 
(rec) -ssh-rsa                              -- key algorithm to remove 
(rec) -diffie-hellman-group14-sha1          -- kex algorithm to remove 
(rec) -hmac-sha1                            -- mac algorithm to remove 
(rec) -hmac-sha1-etm@openssh.com            -- mac algorithm to remove 
(rec) -hmac-sha2-256                        -- mac algorithm to remove 
(rec) -hmac-sha2-512                        -- mac algorithm to remove 
(rec) -umac-128@openssh.com                 -- mac algorithm to remove 
(rec) -umac-64-etm@openssh.com              -- mac algorithm to remove 
(rec) -umac-64@openssh.com                  -- mac algorithm to remove 

# additional info
(nfo) For hardening guides on common OSes, please see: <https://www.ssh-audit.com/hardening_guides.html>

If I had the chance, I'd at least untick the checkboxes from the default config according to the "algorithm recommendations" part, as I only use up-to-date OpenSSH clients.

[AdSchellevis]

The options should be good to add indeed, we don't change any of these defaults currently, so it shouldn't be a huge problem to add them. A test tool is not something we consider for core, if someone wants to build a plugin that's fine, but out of our scope.

[AdSchellevis]

https://github.com/opnsense/core/commit/5df590cd13e34f430fc0be8649550211d8ec38ee should do the trick, install on the latest OPNsense (20.1.2) using:

opnsense-patch 5df590c
service configd restart

[immanuelfodor]

Wow, this was fast, and it's fantastic! Now, the ssh-audit reports are almost fine.

Just two remarks:

1. The ordering of these settings matter, what SSH offers first, second, etc. and even if I tried to tick them in my preferred order, the generated config doesn't respect that. The audit tool doesn't care about the order, it only enumerates them, but the SSH connection's speed, the CPU usage, and even the level of security can be affected (e.g., group18 is very slow but highly secure, curve25519 is fast but still a good tradeoff to put it first). I could have thought of it before, checkboxes don't preserve their ticking order, only the data what is ticked. Maybe if we could store the ticking order in a hidden field, then apply it when generating the config, or have the reorder capability in a popup or something like the firewall rules. What do you think, what could be the best option here? The field itself put the selected items in a comma separated list, this could save the ordering.

https://www.openssh.com/txt/release-5.7 ssh(1)/sshd(8): add a KexAlgorithms knob to the client and server configuration to allow selection of which key exchange methods are used by ssh(1) and sshd(8) and their order of preference.

2. As we offer an ECDSA host key, the ecdsa-sha2-nistp256 KEX algo can't be turned off, it's implicitly on by sshd. Could we have another checkbox field to choose what host keys to offer for clients?

The generated file's bottom part for reference (/usr/local/etc/ssh/sshd_config):

KexAlgorithms diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha256,curve25519-sha256,curve25519-sha256@libssh.org
Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
MACs hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,umac-128-etm@openssh.com
HostKey /conf/sshd/ssh_host_rsa_key
HostKey /conf/sshd/ssh_host_ecdsa_key
HostKey /conf/sshd/ssh_host_ed25519_key

[AdSchellevis]

I think we had this question before, so I though to take a look now, but if the current state isn't useful enough, we might need to revert and see if someone else want to spend time on this.

1. Select boxes can't be ordered, if ssh offers a preferred ordering itself, we can use it, other options currently aren't available

2. We could disable the ones not valid if we can easily map them, include happens here https://github.com/opnsense/core/blob/5df590cd13e34f430fc0be8649550211d8ec38ee/src/etc/inc/plugins.inc.d/openssh.inc#L185-L191

[immanuelfodor]

For the 1st point:

@jtesta might see this thread, he is the maintainer of the mentioned ssh-audit tool, and has a far greater knowledge about SSH internals than me.

• Do you know about any way to get a preferred order out of SSH, rather than than using a seemingly random order from a web settings UI? (I think there is no way, it depends on the user to define their list in the first place, they just use what's given in the config)
• Which one would you choose, a definitely weaker default built-in config or a random order of hardened/selected kex, cipher and mac list? Note that this feature is definitely for those who know what they're doing here. (I think the random order of more secure algorithms is still better than offering weak algos, so I'd not revert the code change done above by @AdSchellevis.)

---

As for the 2nd point, it's great news, so the disabled host key file could be renamed to something else, and the loop continues. When enabling, just rename it back (e.g., ssh_host_ecdsa_key <-> ssh_host_ecdsa_key.bak).

[AdSchellevis]

@immanuelfodor No need to rename, only stop including when not needed, however we do need to determine the logic/mapping between kex algo and hostkey, you can probably help out there.

[immanuelfodor]

The best thing about mapping the algo and the host key is that if the user doesn't include any algo that can be used with the host key, SSH seems to include at least one, so the user won't lock itself out. This happened above when I couldn't get rid of ecdsa-sha2-nistp256 as it was automatically loaded by sshd upon seeing that no chosen algo is preferred for this host key.

With only curve25519 selected on the web interface:

Aaaand it seems I was wrong above, it was not in the kex algo list but the host key algo, and that is not adjustable by the user at all, completely depends on the loaded host key files. 🎉 So no mapping is needed at all.

[immanuelfodor]

Note: I think, here I read kex instead of key before when it was only a two items list with the hardened setup trial.

[immanuelfodor]

Well, after further digging, it seems there is a separate config key entry that I didn't know of, HostKeyAlgorithms, and this is what gets pre-populated by sshd according to the loaded files if not specified: https://www.ssh.com/manuals/server-zos-admin/62/ConfiguringHostKeyAlgorithms.html

And here I could turn off the outdated ssh-rsa algo to comply with the ssh-audit tool by not listing it as my preferred algo. This could be added as the final fourth checkbox field that is in paralel with the four blocks of the ssh-audit output:

# ssh -Q key
ssh-ed25519
ssh-ed25519-cert-v01@openssh.com
ssh-rsa
ssh-dss
ecdsa-sha2-nistp256
ecdsa-sha2-nistp384
ecdsa-sha2-nistp521
ssh-rsa-cert-v01@openssh.com
ssh-dss-cert-v01@openssh.com
ecdsa-sha2-nistp256-cert-v01@openssh.com
ecdsa-sha2-nistp384-cert-v01@openssh.com
ecdsa-sha2-nistp521-cert-v01@openssh.com

https://man.openbsd.org/ssh#Q

I wonder, if the ECDSA and GSA host keys are loaded but no algo is preferred for them, what would happen. I'd be happy if it just wouldn't show up anymore, so the preferred list would be for me:

HostKeyAlgorithms ssh-ed25519,ssh-ed25519-cert-v01@openssh.com

This way we wouldn't need to mess with the file loading. Could you please provide a patch for the fourth checkbox field?

[AdSchellevis]

ok, so long story short, you also would like to add the key selection, which glues to the key generation (last part I'll have to check)

[immanuelfodor]

Yes, it just took me some thinking while writing, sorry 😃

[AdSchellevis]

that's no problem, just making sure we mean the same here :) I'll see if I can add the key selection, ordering isn't something I'm planing to solve. Can wee agree to close this after key is in?

[immanuelfodor]

Yes, we are done here after that, the ordering doesn't matter if I pick my favourite supported algos and only one or two from each of that. The upmost hardened config is the best tradeoff between security and support but it can be hardened further where the order doesn't matter at the end. It's great to have these checkboxes available on the web, that's what matter the most.

[AdSchellevis]

@immanuelfodor ok, I think this https://github.com/opnsense/core/commit/116511910dccfd0431f571279935339391be8373 should be it than (looking at ssh-keygen -t's options)

[immanuelfodor]

Perfect, works as expected, now my audit is completely green, thanks! You can close it now if you wish.