trickeydan
commented
4 years ago SOWN: Investigations into Future Authentication
A dossier because that sounds cool.
Vault
Vault is a fairly nice bit of software for storing static and dynamic secrets. It is developed by Hashicorp, and is mostly open source. It is often used in enterprise environments to solve the issue that we have, albeit on a larger scale than us.
It is backed by a Hashicorp database, "Consul". Consul is a shiny new database thing that operates well in clusters. This makes it well suited for HA. Unfortunately it is super complicated to configure and requires at least three nodes in the cluster, with a quorum of 2 in 3. This makes it very hard to implement for SOWN.
Home-brew Solution
After extensive searching, there doesn't seem to be a tool on the market that does what we need. :disappointed:
We therefore propose a three-part solution to this problem:
- SSH Host Key CA
- SSH User Key CA
- GNU pass to store IPMI, root passwords
It is concluded that we should attempt to add OpenSSH Server to nodes instead of Dropbear in future firmware updates. It is likely out of scope until node_control replacements are being planned and developed.
The following makes multiple references to a new "shell" server, which should be used instead of auth2.
SSH Host Key CA
SSH Host Keys should be signed so that we can validate that we are not SSHing into untrusted hosts. This also removes the need to manually add hosts to known_hosts on the backup servers.
We can store the CA private key in the ansible repo, encrypted with a passphrase. Adding it via ssh-agent might work, Tim to play and test.
Then we can have an ansible playbook generate new host certs with an expiry of 5 years (???), and replace them when they come within a year of expiry.
SSH User Key CA
We can use an SSH User Key CA to allow granular access to hosts.
We will need some custom software to be able to issue certs based off another authentication method. This will also allow us to issue ephemeral keys and certificates for making Ansible deployments.
Certificates that are issued with principals for all machines should be limited to maximum lifetime of 4 hours.
Automated processes such as Ansible, backups and monitoring can request certificates from the authentication server.
A revocation list will need to be distributed to all servers when a new revocation is made.
GW and GW2 should ideally be configured as jump hosts only for all users.
Individual user accounts are only available on shell servers (and possibly GWs (jumping) / Backup Servers (not trusting this with backups)). All other servers have two user accounts:
user- No privileged access.admin- Passwordless Sudo.
So an example of how to get to a certain server would be:
ssh -i ~/.ssh/id_ed25519 -i ~/.ssh/sown_cert.pub -J hafw.sown.org.uk login.sown.org.uk
This can be easily automated using SSH config files, we can generate these files using netbox.
GNU Pass
GNU Pass can be used to store secrets.
It should be stored on shell servers, with a PGP key owned by root.
Other trusted users can also have their PGP key added to the keyring.
The git repo can be automatically pushed to Github.
We should do this so we can have sensible access control to servers.
One other thing that comes to mind, it would be nice if we could automatically set git environment variables. At the very least, we should write some ansible to accept them as forwarded environment vars probably?
Could probably also document a standard ssh_config for proxyjumping etc.