aws-ia / terraform-aws-mwaa

Terraform module for Amazon MWAA(Apache Airflow)
https://registry.terraform.io/modules/aws-ia/mwaa/
Apache License 2.0
41 stars 54 forks source link

Enable startup script #29

Closed easontm closed 1 year ago

easontm commented 1 year ago

What does this PR do?

This PR adds support for the startup script options (#28) now available in the mwaa_environment Terraform resource. It also has some minor aesthetic modifications.

Motivation

I would like to use the startup script in my existing MWAA environments.

More

For Moderators

Additional Notes

To test this, I created a simple startup.sh script whose entire contents was

export WHOS_THERE=TYLER

and uploaded that to S3. I then deployed my forked version of this provider to AWS, using the startup script upload path and the version fetched from the AWS S3 console page. In my IAC folder I swapped the fork in-place (locally, I kept it in aws-ia-mwaa) with the remote version I was using before.

module "mwaa" {
  #  source  = "aws-ia/mwaa/aws"
  source = "../../aws-ia-mwaa"
  #  version = "0.0.1"

  # bunch of normal declaration stuff like name, airflow_version, etc

  startup_script_s3_path = "startup.sh"
  startup_script_s3_object_version = "redacted because I'm not sure if it matters"

I then uploaded a DAG to check the contents of the variable to see if it was exported

from os import environ

from airflow.decorators import dag, task
from datetime import datetime

@dag(
    schedule=None,
    start_date=datetime(2023, 4, 1),
    catchup=False
)
def test_script_dag():
    @task()
    def print_env():
        # Should say TYLER
        print(environ["WHOS_THERE"])

    print_env()

test_script_dag()

And here is an excerpt from the task logs showing that the script I provided via Terraform was correctly executed at environment initialization:

[2023-04-27, 09:14:36 UTC] {{taskinstance.py:1590}} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test_script_dag
AIRFLOW_CTX_TASK_ID=print_env
AIRFLOW_CTX_EXECUTION_DATE=2023-04-27T09:14:30.723200+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-04-27T09:14:30.723200+00:00
[2023-04-27, 09:14:36 UTC] {{logging_mixin.py:137}} INFO - TYLER
[2023-04-27, 09:14:36 UTC] {{python.py:177}} INFO - Done. Returned value was: None
jessicabowden commented 1 year ago

@vara-bonthu when will this be released? 🙏