Staging environment

[sfbetz]

We require a staging environment that is much closer (identical?) to our production environment so we can effectively test new releases before they go live.

[mbarnett]

Brain dumping actions related to this for us to consider when fleshing this out in the new year. Goals here are to make deployments more predictable by better surfacing issues prior to production, and raising visibility of changes to Production configuration so that the Dev team can participate in peer-review of them:

• Target a Staging server set up as close to Production as possible, data and config-wise.
• Rationalize our use of Rails environments (current test.rb serves as both the automated testing config environment and doubles as a quasi-staging environment -- these should be separated. Generally speaking the Rails test environment is only intended for automated testing).
• Get away from rewriting executable code in Production deployment scripts (development.rb/test.rb/production.rb, the sufia.rb initializer, etc) by properly isolating configuration data in non-executable configuration files (secrets.yml, etc). If we have code that runs, it should run on developers' machines and on Plano, not only in Production.
• Ensuring that the ansible playbook that runs on the staging environment is as close to identical to the one that runs on Production as possible.
• Clean up -- or at least thoroughly document and effectively circulate that knowledge outside of the Ops team -- the ansible development repo structure so that the Dev team can better understand what the Ops team is doing with ansible & effectively participate in the process. Without Tricia here I think the rest of us are a bit out of the loop on anything that happens in ansible.
• Leverage tools like GitLab feature branches and Pull Requests during development of ansible changes so that the Dev team can get visibility into/peer-review changes -- this would really help effectively catch issues like the inadvertent change to modsecurity settings that made it into Production.