This PR adds support for the startup script options (#28) now available in the mwaa_environment Terraform resource. It also has some minor aesthetic modifications.
Motivation
I would like to use the startup script in my existing MWAA environments.
More
[x] Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
[x] Yes, I ran pre-commit run -a with this PR
For Moderators
[ ] E2E Test successfully complete before merge?
Additional Notes
To test this, I created a simple startup.sh script whose entire contents was
export WHOS_THERE=TYLER
and uploaded that to S3. I then deployed my forked version of this provider to AWS, using the startup script upload path and the version fetched from the AWS S3 console page. In my IAC folder I swapped the fork in-place (locally, I kept it in aws-ia-mwaa) with the remote version I was using before.
module "mwaa" {
# source = "aws-ia/mwaa/aws"
source = "../../aws-ia-mwaa"
# version = "0.0.1"
# bunch of normal declaration stuff like name, airflow_version, etc
startup_script_s3_path = "startup.sh"
startup_script_s3_object_version = "redacted because I'm not sure if it matters"
I then uploaded a DAG to check the contents of the variable to see if it was exported
from os import environ
from airflow.decorators import dag, task
from datetime import datetime
@dag(
schedule=None,
start_date=datetime(2023, 4, 1),
catchup=False
)
def test_script_dag():
@task()
def print_env():
# Should say TYLER
print(environ["WHOS_THERE"])
print_env()
test_script_dag()
And here is an excerpt from the task logs showing that the script I provided via Terraform was correctly executed at environment initialization:
[2023-04-27, 09:14:36 UTC] {{taskinstance.py:1590}} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=test_script_dag
AIRFLOW_CTX_TASK_ID=print_env
AIRFLOW_CTX_EXECUTION_DATE=2023-04-27T09:14:30.723200+00:00
AIRFLOW_CTX_TRY_NUMBER=1
AIRFLOW_CTX_DAG_RUN_ID=manual__2023-04-27T09:14:30.723200+00:00
[2023-04-27, 09:14:36 UTC] {{logging_mixin.py:137}} INFO - TYLER
[2023-04-27, 09:14:36 UTC] {{python.py:177}} INFO - Done. Returned value was: None
What does this PR do?
This PR adds support for the startup script options (#28) now available in the
mwaa_environment
Terraform resource. It also has some minor aesthetic modifications.Motivation
I would like to use the startup script in my existing MWAA environments.
More
pre-commit run -a
with this PRFor Moderators
Additional Notes
To test this, I created a simple
startup.sh
script whose entire contents wasand uploaded that to S3. I then deployed my forked version of this provider to AWS, using the startup script upload path and the version fetched from the AWS S3 console page. In my IAC folder I swapped the fork in-place (locally, I kept it in
aws-ia-mwaa
) with the remote version I was using before.I then uploaded a DAG to check the contents of the variable to see if it was exported
And here is an excerpt from the task logs showing that the script I provided via Terraform was correctly executed at environment initialization: