Proper, Automated Staging/Release Process

[AlexGilleran]

Problem description

Right now we're with dev config which isn't great.

Ideally we would have something that:

• Clones the production postgres instance
• Snapshots the production elasticsearch instance
• Sets up a staging namespace which runs with production settings except for the loadbalancer ip address, connected to the newly cloned postgres and pulling in the ES snapshot.

and on shutdown

• Kills the cloned DB, being VERY VERY careful not to kill the prod one (the service account that does this shouldn't have permission to do anything to the prod db other than clone it!)
• Kills the whole cluster, as per the preview clusters.

[aneesha09]

Automation is good but do we need 3 staging environments? This might become important if we get obscured/curated datasets to do our testing on. Automation might help us build the environment quickly when needed rather than keeping it live all the time