Document evolving thoughts on staging and production instances

[hancush]

Documentation request

In the past few weeks, we've had to reckon with the tension between our theory that staging sites should be volatile and our practice of using staging sites to create content and data that is then promoted to production.

We would like to align our practices more with the theory of volatility. This has important implications for when we create a production database, and the resources we configure (or don't) for staging.

Loose notes on changes:

• Staging sites should not be used to create content. When a client is ready to begin drafting real content, a production instance, with a production-level database that takes backups, should be created. Consider adopting a practice of flushing the staging database at a regular interval to enforce the volatility principle.
• Staging sites need not necessarily be configured to retain user uploads. Where possible, use the local storage back end rather than configuring an external file store for staging. Uploads will be lost when dynos restart, i.e., once a day, as well as with each new deployment. For instances where there is not acceptable, we should have one and only one security policy and IAM user for staging deployments and permit that user access to S3 buckets tied to staging instances. Production sites should have their own IAM user and policy with access to only the production S3 bucket.

Create or expand the appropriate documentation, with links to helpful configuration (like the one that checks for AWS creds and falls back to local storage if they aren't provided).

[hancush]

1. Research drafting workflow (Wagtail)
2. Wagtail/Django setting to fall back to local storage if S3 not configured (related to #143)
3. Create S3 user/policy for use by staging sites, if needed

[hancush]

4. Strategy for auto-flushing staging databases at regular interval to enforce volatility

[hancush]

Helpful docs on how to create security policy, IAM user, and S3 bucket c/o @smcalilly: https://gist.github.com/smcalilly/2a14731e3075a27de6dc36cf0e75d1e1

[smcalilly]

We've separated staging and production on the same resources for the Agenda Watch project. We're using the same Postgres instance with separate staging and production databases within that instance. We're also doing this with Elasticsearch indexes.

This works fine but it requires the code base to have knowledge of the environment. This makes it harder for me to reason about when configuring builds and writing the code to work with the build so it will use the desired "environment". This was a tradeoff we made to reduce hosting costs, since it costs more to spin up completely separate instances per environment.

Sometimes this tradeoff is necessary, but in most cases a code base should have no knowledge of an environment.