outerbounds / terraform-aws-metaflow

Deploy production-grade Metaflow cloud infrastructure on AWS
https://registry.terraform.io/modules/outerbounds/metaflow/aws/latest
Apache License 2.0
56 stars 48 forks source link

Add custom output path for the DB migrate lambda #47

Closed tgvashworth closed 10 months ago

tgvashworth commented 1 year ago

We had a problem where we were seeing Terraform config drift and unstable plans because the db_migrate_lambda.zip file was being written inside the terraform module folder, which can be outside the repo folder (e.g. if TF_DATA_DIR is set). That means there is no zip file (or index.py) available to a fresh plan and so TF detects a deletion.

A solution to this is to commit or to cache the zip file across builds, but this TF module does not allow that location to be customised.

This PR makes two changes:

  1. The contents of the migrate lambda Python file are now inlined, which makes for one fewer files to manage
  2. There's a variable that specifies the output location so it can be written to the repo/cache location

To test this:

tgvashworth commented 1 year ago

@oavdeev Hey! Do you think someone will be able to have a look at this soon?

tgvashworth commented 1 year ago

Following up on this: we are using a fork of the TF config with this commit, and the lambda etc is all working fine.

leogargu commented 10 months ago

This would fix the perpetual diff we see in our pipelines! ✨ Can it be merged please? Happy to help if needed.

oavdeev commented 10 months ago

Merged as #81