Configure production and staging appropriately

[acozine]

As part of our response to a recent nginx outage, we discovered how easy it is to run commands on a production machine while believing you are logged into a staging machine. We have two goals in configuring our two environments:

1. Make production and staging work as similarly as possible, to support "muscle memory" of commands and to make sure that when we test changes in staging we have a realistic preview of how those changes will affect our production systems. We do fairly well at this already and we don't want to lose that.
2. Make production and staging look and possibly respond differently enough that we are aware of which environment we are logged into at any time.

We have already added a banner to production machines, but that only appears on first login. Can/should we do more? We have a wide range of options for making production noticeably different from staging, including some fairly draconian changes. For example, we could:

• On production, require a password for sudo actions
• On production, respond to sudo commands with a "this is production, are you sure?" message
• On production, respond to all commands with a "this is production, are you sure?" message
• On production, disallow SSH access, require all changes to be run by automation

Let's add more options, discuss/debate, and come up with acceptance criteria for this ticket.

[tpendragon]

It looks like one option is a little sudo script that says "Are you sure?" and then delegates out to /bin/sudo