Enable Django to use automated testing using secrets downloaded from S3 config bucket

[MikeTheCanuck]

Design Goal and Approach

One of the fundamental design goals of the current devops squad's automation pipeline is to support multiple target environments for the backend API builds. At present we have plans to support two target environments: "integration" (previously known as staging) and "production". [Unclear at this time exactly what condition would cause the build pipeline to configure for "integration/staging" vs "production" database, or to deploy a successful container image to "integration/staging" vs "production". Perhaps it keys off of which branch in the project's repo is the commit target; perhaps it's some other condition.]

To support the ability to switch among multiple secrets (e.g. DATABASE_PASSWORD) that vary depending on the target environment (i.e. we want to define a different database password for the PostgreSQL instance in "integration" vs the instance in "production"), we have decided to use the same environment variables in the project's code, and to pull in different values for those env vars from outside files.

The outside files are stored in a secured AWS S3 bucket, and the project's Travis repo will be configured with AWS creds to read the files.

Travis will download the appropriate file depending on target environment e.g. download /integration/env.sh for the "integration" environment, or download /production/env.sh for the "production" environment.

The contents of the file will be imported into the build such that the secrets stored in the file will be available to Django and to Travis commands at build time. e.g. in env.sh, the command export DATABASE_PASSWORD=string emits the env var to Travis build environment, to be used by settings.py using the statement os.environ.get("DATABASE_PASSWORD").

Current Issues

I have attempted to incorporate either approach into a Dockerized Django app that builds automatically through TravisCI. Both approaches have consistently failed for one reason or another.

The issues I've encountered, and the experiments I've run to address them, are documented here:

• ImportError: cannot import name 'project_config'
• Enable functional testing of API container in Travis using "source env.sh" approach

Current workaround

The current team-budget backend Travis build has temporarily forgone the use of externally-derived env vars, and just hard-coded them into the Travis repo settings. Not pretty, and we intend to change that in the future, but for the moment it enables us to use Travis' build outcomes to validate the enhancements being submitted by each developer.

[pdxdiver]

See the https://github.com/hackoregon/backend-service-pattern

[MikeTheCanuck]

The test requirement has not been addressed in backend-service-pattern.

Since it appears that GitHub doesn't allow the author to re-open a Closed issue, I'm going to have to open a new issue to track the testing issue.

[pdxdiver]

It's fixed. Will re-open till it's validated by you

[MikeTheCanuck]

Thanks Dan.

I have zeroed in on a solution to this for the Budget backend over the past 24 hours, and validated it against the pattern that Dan has implemented in the backend-service-pattern.

See Budget team PR 74 for details.