Deploy on AWS

[donnell794]

The current plan is to deploy to AWS Beanstalk which will take care of scaling, HTTPS, and blue-green deploy (zero-downtime deploy), all while still giving us access to services like RDS.

Tasks

• [x] Remove ansible playbooks.
• [x] Create Terraform templates to create the beanstalk application and RDS database.
• [x] Create shared Terraform state on S3 for shared access.
• [x] Configure CI to continuously deploy the application to the beanstalk instance.

[adborden]

Updated the description to capture the current plan to use Beanstalk and Terraform.

[adborden]

I've got the Terraform templates working. Initially, I wanted the terraform templates to live here, but now I'm thinking it makes sense to live in https://github.com/openoakland/infra.

Originally, I wanted the Terraform templates to be applied automatically via CI/CD, but the Terraform + CircleCI combo isn't great at keeping secrets secret. Terraform has to output some secrets into the plan/logs so a human can review them. So given we're using a public CI system, secrets will be leaked in the logs/artifacts to the public. So for that reason, we'll have to run terraform as a one-off manually and will be run less frequently. So it's not really connected to the rest of the OakCrime project and makes more sense to live in OpenOakland's infrastructure repo.

[rbelew]

"... makes more sense to live in OpenOakland's infrastructure repo" this mostly sounds right to me, making it part of the OO provisioning process and localizing access to secrets. but on the other hand it seems like it requires whomever doing this for OO to have more insight into the OakCrime project than they probably want. knowing nothing about Terraform templates... is there a way to have some minimal set of them that could live on the OO side, referencing more detailed OakCrime-specifics templates in our repo?
[i'm also trying to anticipate other projects wanting to follow a similar deployment trajectory.]

[adborden]

@rbelew these are great points and I'm looking at the Infra repo and still thinking about how it all could work together.

The terraform setup actually doesn't require much knowledge about OakCrime (I can walk you through the details about why that is offline). The terraform templates in this case are focused on a docker-based deploy, it assumes you're a web application and need a database. It does assume a few environment variables specific to Django/OakCrime (e.g SECRET_KEY, EMAIL_SERVER) but we can move this out so it assumes nothing about Django/OakCrime. So they are fairly reusable already, but need would a little more work.

This gives me another idea though, maybe OpenOakland should supply reusable Terraform modules for doing things the "OpenOakland way", and then projects would host their own templates by pulling in the modules.

@tdooner do you want to weigh in here?

[adborden]

... and now I realize I'm contradicting myself a little.

It does assume a few environment variables specific to Django/OakCrime (e.g SECRET_KEY, EMAIL_SERVER) but we can move this out so it assumes nothing about Django/OakCrime

So even though we can make these variables agnostic to OakCrime, they do live in terraform. Which means if you were to wanted to change or add an environment setting, then the change would be made first in the OakCrime repo and then in the terraform repo. So having these in different repos is not ideal.

If we went with having a separate repo for the terraform modules, the terraform code could still live in OakCrime, so all the OakCrime bits are in a single place.

[tdooner]

I've never used Elastic Beanstalk so I may have no idea what I'm talking about here. But, is it possible for Terraform at the OpenOakland level to do the following things?

• Create the Elastic Beanstalk app
• Create any auxiliary AWS resources required (database, load balancer, networking, etc.) that Elastic Beanstalk doesn't provide
• Create IAM users for the team to do the deploys/management of the Elastic Beanstalk

And then the people working on the project would have IAM users that are able to update configuration settings (the SECRET_KEY env var, etc.), and deploy new versions of the code?

I think it is okay for us to have Terraform run as a one-off (assuming we can figure out how to do locking, shared state management, etc.) by a small OpenOakland "infra" team, but only in the case where the actions that would require us to do a Terraform run are the project lifecycle events -- adding or removing a team member or project -- rather than day-to-day events like deploying a new version of the code or the code's configuration.

[adborden]

Yes, we can do all of those things! (and in fact it does). The beanstalk branch has the terraform code, so you can take a look.

Conceptually, Elastic Beanstalk is like Heroku, except it provides an eb command to do the deploy. So on that branch, you'll see we create a CI user (oakcrime-ci) and run eb deploy via CI. The terraform templates create the application, and a single production environment (staging could be added later if needed).

This is the main terraform file. The only bits that feel like maybe I'd want it in the OakCrime repo is the environment variables. If you wanted to change or add an enviornment variable, you'd need to do it from the Infra repo. You could do it from an IAM user, but it would be overwritten the next time the Infra templates are executed.

We had a discussion earlier about using env-specific settings files in code vs. specifying configuration via environment variable. We landed on the environment variable path but having env-specific settings files in code allows us to keep almost everything within OakCrime. Only secrets would live outside of OakCrime and they would need to live outside of code anyway. Sounds like we should re-consider that decision.

[rbelew]

real progress @adborden !

the push to all-environment variables vs. within code was recent and designed to simplify porting, so i'm happy to stick them back (in settings.py). but i'm not clear how/where you imagine the secrets getting included?

[adborden]

On Mon, 2019-02-04 at 09:49 -0800, Rik Belew notifications@github.com wrote:

real progress @adborden !

the push to all-environment variables vs. within code was recent and designed to simplify porting, so i'm happy to stick them back (in settings.py). but i'm not clear how/where you imagine the secrets getting included?

The secrets will exist how they are today. They will be stored offline by Infra users and provided to the Django application via environment variable from the beanstalk stack.

So for example, SECRETS_KEY will live in terraform, the actual value will be stored offline. It can be provided to terraform at runtime via environment variable (TF_VAR_django_secret_key) or could be put in a secrets.tfvars file. Whoever runs terraform would need those offline secrets but any OakCrime developer would never need them directly.

I can walk you through this offline if you'd like.

-- Aaron D Borden Human and Hacker