Encourage sites to configure /etc/ssh/ssh_known_hosts for login and submit hosts?

[ericfranz]

The LinuxHostAdapter and ssh wrapper features both include setting strict_host_checking: false for when the host is not in the user's ~/.ssh/known_hosts file. It seems that we should at least document, or even provide a helper script, to:

1) add the hosts OnDemand is configured to ssh to in the /etc/ssh/ssh_known_hosts file 2) some type of verficiation script to run to output the hosts that need to be added when cluster configs are modified to specify a new login or submit host that is not in /etc/ssh/ssh_known_hosts

I do not know if it is /etc/ssh/ssh_known_hosts is always the path, or if sometime it could be /etc/ssh/known_hosts

┆Issue is synchronized with this Asana task by Unito

[treydock]

This all sounds extremely fragile. I would recommend using these SSH flags:

-oStrictHostKeyChecking=no -oUserKnownHostsFile=/dev/null -oLogLevel=ERROR

We use these on clusters for both clustershell and pdsh, so that the only thing looked at for SSH authorization is /etc/ssh/ssh_known_hosts. The default also is /etc/ssh/ssh_known_hosts but that value can be changed.

I would not recommend adding code to OnDemand to touch the global known hosts file. I think we just need to document the need keep that file up-to-date. This is something a site's automation should be handling, NOT OnDemand. Also for clusters it's usually common for sites to have a single host key for the entire cluster and use aliases to match all hosts...this is alias for Owens:

owens-rw01.ten.osc.edu,o0*,owens-rw*,owens-login*,owens.hpc.osc.edu,owens.osc.edu,10.2.*,!10.2.200.*,ib-o0*,ib-owens-rw*,ib-owens-login*,10.22.5?.*,10.22.6?.*

We would be imposing ourselves a huge burden to try and meet all use cases of SSH known hosts if we start touching the file. I think we need to leave it to sites to handle this otherwise it will become something we will never stop updating to catch up to all the ways sites use SSH host keys. So I think documentation is all we need to do.

[treydock]

Also it's really important to use UserKnownHostsFile=/dev/null because if a host key changes it will prevent entries not managed by automation from causing conflicts. It's best not to read user's known hosts file to keep things simple and also prevents the user's file from having new entries automatically added and thus relying only on the global known hosts file.

[ericfranz]

Is there any security concern for the Linux Host Adapter and the ssh wrapper for submitting jobs to both default to StrictHostKeyChecking=no? I had assumed that this would be a type of "insecure default" so should be opt-in i.e. manually specify strict_host_checking=false in the cluster config, with it defaulting to true.

Also it seems you are suggesting that even if strict_host_checking=true we would still do UserKnownHostsFile=/dev/null and expect the global known hosts file to be properly configured. Is that correct?

[treydock]

Yes, setting StrictHostKeyChecking=no is technically an insecure default so maybe we need to provide a means to allow sites to specify their own SSH flags with some sane defaults provided like UserKnownHostsFile=/dev/null.

Yes we need UserKnownHostsFile=/dev/null otherwise every time a user logs in with SSH for first time the host key will get added to ~/.ssh/known_hosts and also that file will be checked for matching host key. If the host key is changed and the host's old key exists in ~/.ssh/known_hosts it will cause errors.

One thing I vaguely recall is if you set UserKnownHostsFile=/dev/null then every SSH login will prompt for yes/no to accept unless you set StrictHostKeyChecking=no so it might be the case that we need to document the "insecure" default and make sites aware of this. If we are concerned about a potentially insecure default then we would need a way to allow sites to provide SSH flags and we would need to document the recommended "insecure" defaults. FWIW I believe that anyone putting their hosts into LinuxHostAdapter will trust those hosts so StrictHostKeyChecking=no is not necessarily insecure as it's not likely a site is running OnDemand at SiteA and then setting up LHA to SSH to a external entity's site. I like to think that most security comes down to common sense and while at first glance StrictHostKeyChecking=no seems insecure, in practice the way it's used with LHA is not insecure.

[ericfranz]

Testing from my mac, if I do

ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=yes owens.osc.edu

or

ssh -oUserKnownHostsFile=/dev/null owens.osc.edu

After doing

mv .ssh/known_hosts .ssh/known_hosts.bak
sudo cp .ssh/known_hosts.bak /etc/ssh/ssh_known_hosts

So if the global /etc/ssh/ssh_known_hosts is configured, we can set UserKnownHostsFile=/dev/null and have strict host checking enabled as a secure default.

[treydock]

I'd recommend testing on Linux, the behavior maybe different. If the entry is in /etc/ssh/ssh_known_hosts and you do /dev/null for the user file, I think you are correct that no need to add StrictHostKeyChecking...now that I think about it more I do recall that's the behavior I see when adding new hosts at OSC and once I add the host to bastion SSH known hosts the user /dev/null is fine and doesn't prompt yes/no.

[ericfranz]

The ssh man pages on Owens are the same as on my Mac in this regard so I’d be surprised if the behavior differed. But will check

[matthu017]

Testing on Fedora Linux, the command ssh -oUserKnownHostsFile=/dev/null -oStrictHostKeyChecking=yes owens.osc.edu worked as intended after moving known_hosts to /etc/ssh/ssh_known_hosts. The exact commands can be found above in Eric's comment.