search

datarootsio / terraform-aws-ecs-airflow

A terraform module that creates an airflow instance in AWS ECS.
MIT License
59 stars 41 forks source link

Issue 15/sync dag #16

Closed biermansm closed 3 years ago

biermansm commented 3 years ago

Modify the s3 sync DAG to use the awscli sync function.

Resolves #15

biermansm commented 3 years ago

I deployed this version for one of our clients and it seems to work, but I wonder if anyone else could verify this as well? Might be that it installed awscli through our requirements.txt file instead of init (but I hope not).

nicogelders commented 3 years ago

you put the pip install in the init container, it had to go in the webserver/scheduler container. The rest looks good thanks! Will test it out and merge it later this week!