feat: document validator best practices

[jcstein]

From slack:

Types of setups

• active + backup node with tmkms
• 3 sentries setup with tmkms/horcrux
• a mix of validator + 1-3 sentries + horcrux/tmkms

Best practices

Validator security best practices:

• do not manage your node from root user, disable root password
• use ssh keys (preferably something like Yubikey) to login to your server
• use ufw, iptables, firewalld etc. to limit access to only necessary ports and IP addresses
• use intrusion prevention software like fail2ban
• setup DDoS protection via services like Cloudflare
• use encryption-enabled secure storage for sensitive data (private keys, passwords, node keys etc.) to protect them from unauthorized access
• always backup your node keys, eg. priv_validator_key.json and node_key.json and store them securely
• use hardware wallets or authz module to secure your validator wallet
• use hardware security models (HSMs) to store and manage private keys
• use tools like TMKMS and Horcrux for your validator signing key: TMKMS with YubiHSM: https://github.com/iqlusioninc/tmkms/blob/main/README.yubihsm.md, tmkms: https://github.com/iqlusioninc/tmkms/tree/main, horcrux: https://github.com/strangelove-ventures/horcrux/blob/main/docs/migrating.md
• always use at least a backup node for you validator, in case the primary goes down, or sentry nodes
• use monitoring systems like Prometheus/Grafana/Tenderduty along with real time alerts like PagerDuty some articles as well: https://dydx.forum/t/sharing-infrastructure-best-practices-for-running-validators-on-cosmos-based-chains-like-dydx/2318

To do

• [ ] add a validator best practices page

[jcstein]

Link to cosmos docs where necessary: https://docs.cosmos.network/main/user/run-node/run-production

[GeoddHQ]

@jcstein we plan to implement test active + backup node with tmkms on testnet to help you guys validate the best possible setup. I will share my journey here as we progress