Terraform module to deploy an Apache Airflow cluster on AWS, backed by RDS PostgreSQL for metadata, S3 for logs and SQS as message broker with CeleryExecutor
Apache License 2.0
84
stars
40
forks
source link
Issues deploying an Airflow cluster due to obsolete ami image default and environment provisioning failure in EC2 instances. #37
The default ami image used to deploy EC2 instances is no longer available (at least in eu-west-1 zone), I must use the more recent: ami-089cc16f7f08c4457" # ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200611, instead setting it in main.tf with "ami =" option without problem.
In EC2 instance provisining file "cloud-init.sh" I got a failure setting the Airflow environment variables in /etc/environment due to incorrect line format in file because of the use of preceding "export ", to avoid that I propose the next patch:
--- .terraform/modules/airflow_cluster/terraform-aws-airflow-0.12.0/files/cloud-init.sh.orig 2020-06-25 11:42:51.153032110 +0000
+++ .terraform/modules/airflow_cluster/terraform-aws-airflow-0.12.0/files/cloud-init.sh 2020-06-26 06:42:23.942287388 +0000
@@ -92,11 +92,12 @@
cat /etc/environment | sudo tee -a /tmp/airflow_environment
cat /tmp/custom_env | sudo tee -a /tmp/airflow_environment
- sed 's/^/export /' -- </tmp/airflow_environment | sudo tee -a /etc/environment
+ sudo cp /tmp/airflow_environment /etc/environment
+ sed 's/^/export /' -- </tmp/airflow_environment | sudo tee -a /tmp/airflow_exportenv
sudo cat /tmp/airflow.service >> /etc/systemd/system/airflow.service
cat /tmp/airflow_environment | sudo tee -a /etc/sysconfig/airflow
- source /etc/environment
+ source /tmp/airflow_exportenv
if [ "$AIRFLOW__CORE__LOAD_DEFAULTS" = false ]; then
airflow upgradedb
Hi all and greetings for this great job.
The default ami image used to deploy EC2 instances is no longer available (at least in eu-west-1 zone), I must use the more recent: ami-089cc16f7f08c4457" # ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200611, instead setting it in main.tf with "ami =" option without problem.
In EC2 instance provisining file "cloud-init.sh" I got a failure setting the Airflow environment variables in /etc/environment due to incorrect line format in file because of the use of preceding "export ", to avoid that I propose the next patch:
Best regards,
Juan M.Victoria