Enable Django to perform automated testing using secrets downloaded from S3 config bucket

[MikeTheCanuck]

Design Goal and Approach

One of the fundamental design goals of the current devops squad's automation pipeline is to support multiple target environments for the backend API builds. At present we have plans to support two target environments: "integration" (previously known as staging) and "production". [Unclear at this time exactly what condition we'll used to in the future to cause the build pipeline to configure for "integration/staging" vs "production" database, or to deploy a successful container image to "integration/staging" vs "production". Perhaps it'll key off of which branch in the project's repo is the commit target; perhaps it's some other condition.]

To support the ability to switch among multiple secrets (e.g. DATABASE_PASSWORD) that vary depending on the target environment (i.e. we want to define a different database password for the PostgreSQL instance in "integration" vs the instance in "production"), we have decided to use the same environment variables in the project's code, and to pull in different values for those env vars from outside config files.

The outside config files are stored in a secured AWS S3 bucket, and the project's Travis repo will be configured with AWS creds to read the files.

Travis will download the appropriate config file depending on target environment e.g. download /integration/env.sh for the "integration" environment, or download /production/env.sh for the "production" environment.

The contents of the config file will be imported into the build such that the secrets stored in the file will be available to Django and to Travis commands at build time. e.g. in env.sh, the command export DATABASE_PASSWORD=string emits the env var to Travis build environment, to be used by settings.py using the statement os.environ.get("DATABASE_PASSWORD").

Current Issues

I have attempted to incorporate either approach into a Dockerized Django app that builds and runs tests automatically through TravisCI. Both approaches have consistently failed for one reason or another.

The issues I've encountered, and the experiments I've run to address them, are documented here:

• ImportError: cannot import name 'project_config'
• Enable functional testing of API container in Travis using "source env.sh" approach

Current workaround

The current team-budget backend Travis build has temporarily forgone the use of externally-derived env vars, and just hard-coded them into the Travis repo settings. Not pretty, and we intend to change that in the future, but for the moment it enables us to use Travis' build outcomes to validate the enhancements being submitted by each developer.

[pdxdiver]

See updated backend pattern

[MikeTheCanuck]

I have zeroed in on a solution to this for the Budget backend over the past 24 hours, and validated it against the pattern that Dan has implemented in the backend-service-pattern.

See Budget team PR 74 for details.