Changed host keys are annoying and undermine security → use host certificates

[ePaul]

When running odd in a cloud environment (as it is supposed to), you will regularly have new instances (e.g. after a version upgrade or just because AWS feels it needs to replace the hardware), still being available at the same domain name as the old ones. Those new instances have new host keys, resulting in a warning message like this one:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:INAii/cXbAT/2hovKHLkxToJ7xqQvLPfzvdah3Q5APw.
Please contact your system administrator.
Add correct host key in /Users/name/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/name/.ssh/known_hosts:6
ECDSA host key for odd-eu-west-1.team.example.com has changed and you have requested strict checking.
Host key verification failed.

Of course, a workaround is to disable the host checking (and even disable storing of the host keys), but this has the effect that now we basically do not have any remote host verification, which is an important part of SSH, because it protects against man-in-the-middle attacks (actually, a MitM attack is already impossible with public-key auth, but there still could be a fake server pretending to be the real one).

There should be some automated way of verifying those keys.

A possible solution could be to use certificates confirming the host key's authenticity. I just found this article How To Create an SSH CA to Validate Hosts and Clients with Ubuntu, which explains the process. (It also has client certificates, which are not needed here, because that is the job of even.) I guess we would need a slightly more elaborate process for STUPS than explained there (like each odd instance after startup (and before its validity time runs out) needs to request a certificate from some certification authority server – potentially authorized using OAuth?), but it should serve as a starting point.

With this, all the (human) users just add a @cert-authority line into their .known-hosts file, and never will see such a warning again.

[szuecs]

What do you think if piu triggers a gethostkey to "even" and would take care, that your local known_host is up to date? I don't like the idea to create yet another CA.

[ePaul]

That could be a different way of solving this, I guess. (Not sure if everyone wants piu to modify their known_hosts file, so there should be some CLI option and/or config file setting to enable/disable this).

Does even currently know the odd host's key? (I guess it should, because it is itself connecting there.)

[szuecs]

Good question, we have to look into it.