Determine Best Practice for Deploying Lambda Function without Docker Container

[ben851]

Description

As a developer/operator of GC Notify, I would like to be able to deploy the infrastructure without having dependencies on the application. Currently, we are unable to deploy the environment from scratch because AWS expects the docker image to exist when deploying a lambda function.

Affected components:

• ses_receiving_emails
• heartbeat
• lambda-admin-pr
• lambda-api
• lambda-google-cidr

WHY are we building?

• BCP
• Reliability
• Dynamic Environments

WHAT are we building?

Investigate different options for how to decouple the application from the infrastructure

VALUE created by our solution

We will be able to better maintain our infrastructure, and deploy to new test environments

Acceptance Criteria

• [ ] ses_receiving_emails can be deployed without dependency on the application, or at the least, the dependency is managed via code
• [ ] Production and Staging are able to be migrated to this new model

QA Steps

• [ ] Test against new environment
• [ ] Test against staging/production

[ben851]

My initial thoughts:

1. Separate the ECR repositories from the lambda deployment, but keep them in the same notification-terraform repository. Create terraform post hook to build the docker image, and push it to the newly created repository
2. Create the beginnings of a notify-depot terraform - the infrastructure for the infrastructure. Then we would manage initial scratch deployments via additional pipelines:
   • notify-depot apply
   • build/push docker images
   • notify-terraform apply

[ben851]

• There is a custom module that can build and push docker images but it is expecting the dockerfile to be a path on the system. When I tried using git clone as a pre-hook, the provider constantly crashed.

• I did a PoC with the lambda API to move the ECR to a separate folder and then use null_resources to execute docker commands and build/push the image. It works but is clunky, going to bounce it off of Pat and continue investigation to better ways.

[ben851]

• I have created a new ecr folder within the terraform code that creates null_resources to build and push bootstrap containers.
• I modified the lambda image references to have a customizable docker tag. If no tag is sent, the bootstrap is used. Otherwise, the bootstrap is overridden with the correct tag
• This will require a lot of testing/QA in staging to ensure we don't break pipelines

[ben851]

Currently blocked by task 103 - automate ACM certificate validation process, which is in turn blocked by task #36 move the DNS to notify aws accounts.

[ben851]

• DNS migration done this morning, will PR re-enabling ACM validation today.

[sastels]

waiting for the ACM validation changes

[ben851]

• ACM Validation changes pushed to staging May 16.
• Production release reconciliation May 18
• Will get back to this Friday May 19

[ben851]

ACM validation deployed to Prod. Will dust off my docker PR and work on merging this afternoon.

[jimleroyer]

@ben851 was able to build the whole environment without code changes yesterday. But destroying an environment cause problems as some resources failed to get deleted. Ben will investigate a script to nuke everything if he can't get his ways using Terraform.

[jimleroyer]

We are reviewing that PR at the moment: https://github.com/cds-snc/notification-terraform/pull/764