Move off AWS Lambda / API Gateway

[rufuspollock]

We've had a bunch of issues with Lamba / API gateway setup:

• Intermittent errors on site
• Some issues with zappa
• Forced https and complete PITA setting up CNAMEs etc (and apex domains are even worse)
• Complex logging
• The fact the site is deployed at /staging/ rather than /

=>

• Let's move off AWS Lambda
• Let's use:
  • Longer term: (probably) docker
  • Right now: heroku

Acceptance Criteria

• [ ] No zappa or AWS stuff in the codebase or deploy code
• [ ] All AWS relevant stuff (e.g. lambda) shutdown
• [ ] deploy script works with new setup
• [ ] staging is working (on heroku)

Tasks

Prep

• [x] Audit current AWS assets and identify:
  • [x] Which we can reuse for Heroku
  • [x] Which are for local dev
  • [x] Which we can delete
• [x] Identify any specific zappa dependencies in the code
• [x] Review dpr deploy

Implement

• [x] Update dpr deploy (5h)
  • Looking at this it looks like moving off ansible will be simplest
• [x] Deploy to heroku (1.5h) [4h] (took 3h as we shot ourselves in the foot with a new rds instance config bug!)
  • [x] Put in heroku hooks (Procfile etc)
  • [x] Use existing RDS and S3 (?)
  • [x] Refactor build of dpr-js bundle!
• [x] Remove zappa in dpr-api
  • [ ] FlaskS3Bucket stuff
• [ ] Delete all the AWS stuff
  • [x] Lambda
  • [x] Gateway
  • [x] Cloudfront
  • [ ] S3 bucket for assets

Analysis

graph TD

  user[fa:fa-user User] --> frontend
  frontend[Frontend] --> db[Database]
  frontend --> bits[S3]

AWS

graph TD

  user[fa:fa-user User] --> cloudfront
  cloudfront --> apigateway[API Gateway]
  apigateway --> lambda[AWS Lambda - Flask via Zappa]
  cloudfront --> s3assets[S3 Assets]
  lambda --> rds[RDS Database]
  lambda --> bits[S3 Bits]

Heroku

graph TD

    user[fa:fa-user User] --> heroku
    heroku[Heroku - Flask] --> rds[RDS Database]
    heroku --> bits[S3 Bits]

[rufuspollock]

One issue with the ssl: we can't get cloudflare ssl working with bits.xxx.yyy.com because cloudflare wildcards only cover one level (see this doc).

To fix this: move the staging bits to bits-staging.yyy.com (for full production we could still use bits.yyy.com).

• [x] Set up bits-staging in cloudflare
• [x] configure staging app to use this

[rufuspollock]

90% done but not quite there in this sprint so moving to next.

[zelima]

Still remaining things like remove flasks3bucket stuff and cleaning up buckets on AWS s3

[zelima]

FIXED all the ansible and zappa stuff removed from code base