Support hosting static / media files in S3

[msheiny]

This is an optional deployment setting controlled via environment variables. Of particular use to me was to ensure that we are able to use IAM roles which will mirror deployment so that is what I tested on pretty exclusively.

Here's a good test plan if you want to mimic that locally:

• Rebuild the prod container make build-prod-container

• change the nginx docker config to look like this

  
  diff --git a/docker/nginx/proxy.conf b/docker/nginx/proxy.conf
  index fd6acdf..b250698 100644
  --- a/docker/nginx/proxy.conf
  +++ b/docker/nginx/proxy.conf
  @@ -1,14 +1,6 @@
  server {
   listen 8080;

• location /media {

• alias /django-media;

• }

• location /static {

• alias /django-static;

• }

• location / { proxy_pass http://app:8000; proxy_set_header Host app;

* assume a role using the aws cli and dump those creds to a file .. `aws sts assume-role --role-arn arn:aws:iam::$account:role/$role_name --role-session-name "temp-stn-session" >  assume-role-output.txt`.
* Extract the elements above from the file and set them to `AWS_SESSION_TOKEN`, `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` in `docker/ci.env`
* Set your bucket name in `docker/ci.env` to `AWS_STORAGE_BUCKET_NAME` (it has to be a bucket that the role you provided can write to).
* finally instead of doing the make target to bring the prod env up... doing it manually to watch the logs go by `docker-compose -f ci-docker-compose.yaml up` (you may have to run `docker-compose -f ci-docker-compose.yaml down -v` if you havent run it for a while first)
* lastly make sure the static files got dumped into the bucket as expected under a `static/` path

[msheiny]

rebased on master

[msheiny]

currently adding features for google cloud storage .. do not merge yet :)

[msheiny]

How to test with GCE support:

1. Create a service account with following project level roles > roles/storage.objectViewer, roles/storage.objectCreator, roles/cloudsql.client
2. Create a bucket with following bucket level perms roles/storage.objectViewer, roles/storage.objectCreator, roles/storage.legacyBucketReader 3 . Dump key from that service account and copy inside a new folder called ./creds
3. Apply this diff to ci-docker-compose.yaml

   diff --git a/ci-docker-compose.yaml b/ci-docker-compose.yaml
   index 76247a4..1a19a28 100644
   --- a/ci-docker-compose.yaml
   +++ b/ci-docker-compose.yaml
   @@ -32,6 +32,7 @@ services:
      - ${HOST_STATIC_DIR:-stn-django-static}:/django-media
      - ${HOST_MEDIA_DIR:-stn-django-static}:/django-static
      - ${HOST_LOGS_DIR:-stn-django-logs}:/django-logs
   +      - ./creds:/gce-creds
    networks:
      app:
        aliases:
   diff --git a/docker/ci.env b/docker/ci.env

4. Append to docker/ci.env to look similar to this

   +GS_BUCKET_NAME=______________
   +GS_PROJECT_ID=__________________
   +GS_CREDENTIALS=/gce-creds/${name_of_your_service_account.json}

5. Edit nginx.conf to look like this

   
   diff --git a/docker/nginx/proxy.conf b/docker/nginx/proxy.conf
   index fd6acdf..4a7b948 100644
   --- a/docker/nginx/proxy.conf
   +++ b/docker/nginx/proxy.conf
   @@ -1,10 +1,6 @@
   server {
    listen 8080;

• location /media {
• alias /django-media;
• }

• location /static { alias /django-static; }

  
  6. Run `docker-compose -f ci-docker-compose.yaml up`
  7. Log into the admin `http://localhost:8080/admin/` go to `Images` and upload an arbitrary file
  8. You shouldnt see any errors.. make sure your image got dumped to your bucket :) You can use `gsutil` like `gsutil ls -p ${proj_name} gs://${bucket_name}/`

[msheiny]

@conorsch any objections to merge here?

[msheiny]

hey @harrislapiroff / @ketudb can i get a review here? At the bare minimum you can just confirm it doesn't break anything with the settings disabled.

[msheiny]

Or visual review ;)