aws / aws-mwaa-local-runner

This repository provides a command line interface (CLI) utility that replicates an Amazon Managed Workflows for Apache Airflow (MWAA) environment locally.
MIT No Attribution
672 stars 683 forks source link

AIRFLOW_ENV_NAME #56

Closed KarthikRajashekaran closed 2 years ago

KarthikRajashekaran commented 2 years ago

When I tried running the below command inside the docker python I didn't find the variable AIRFLOW_ENV_NAME

pprint.pprint(dict(os.environ), width=1)

{'AIRFLOW_HOME': '/usr/local/airflow',
 'EXECUTOR': 'Local',
 'FERNET_KEY': '****************,
 'HOME': '/usr/local/airflow',
 'HOSTNAME': '*******',
 'LANG': 'en_US.UTF-8',
 'LD_LIBRARY_PATH': '/usr/local/lib',
 'LESSOPEN': '||/usr/bin/lesspipe.sh '
             '%s',
 'LOAD_EX': 'n',
 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:',
 'PATH': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
 'PWD': '/usr/local/airflow',
 'SHLVL': '1',
 'TERM': 'xterm',
 '_': '/usr/bin/python3'}

Whereas when I run in AWS MWAA we have as below (sample) , I do see 'AIRFLOW_ENV_NAME': 'airflow-dev'

I was expecting the same variable in aws-mwaa-local-runner

[2021-09-29 13:10:32,673] {{taskinstance.py:1283}} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=admin@localhost.com
AIRFLOW_CTX_DAG_OWNER=airflow
AIRFLOW_CTX_DAG_ID=dag_with_task
AIRFLOW_CTX_TASK_ID=print_the_context
AIRFLOW_CTX_EXECUTION_DATE=2021-09-29T13:10:31.098987+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-09-29T13:10:31.098987+00:00
[2021-09-29 13:10:32,693] {{logging_mixin.py:104}} INFO - {'AIRFLOW_CONFIG_SECRETS': '{"AIRFLOW__SECRETS__BACKEND_KWARGS":"{\\"connections_prefix\\" '
                           ': '
                           '\\"airflow/connections\\", '
                           '\\"variables_prefix\\" '
                           ': '                         '\\"airflow/variables\\"}","AIRFLOW__SECRETS__BACKEND":"airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend"}',
 'AIRFLOW_CONN_AWS_DEFAULT': 'aws://',
 'AIRFLOW_CONSOLE_LOGS_ENABLED': 'true',
 'AIRFLOW_CONSOLE_LOG_LEVEL': 'WARNING',
 'AIRFLOW_CTX_DAG_EMAIL': 'admin@localhost.com',
 'AIRFLOW_CTX_DAG_ID': 'dag_with_task',
 'AIRFLOW_CTX_DAG_OWNER': 'airflow',
 'AIRFLOW_CTX_DAG_RUN_ID': 'manual__2021-09-29T13:10:31.098987+00:00',
 'AIRFLOW_CTX_EXECUTION_DATE': '2021-09-29T13:10:31.098987+00:00',
 'AIRFLOW_CTX_TASK_ID': 'print_the_context',
 'AIRFLOW_DAG_PROCESSING_LOGS_ENABLED': 'true',
 'AIRFLOW_DAG_PROCESSING_LOG_LEVEL': 'WARNING',
 'AIRFLOW_ENV_NAME': 'airflow-dev',
'AIRFLOW_HOME': '/usr/local/airflow',
 'AIRFLOW_SCHEDULER_LOGS_ENABLED': 'true',
 'AIRFLOW_SCHEDULER_LOG_LEVEL': 'WARNING',
 'AIRFLOW_TASK_LOGS_ENABLED': 'true',
 'AIRFLOW_TASK_LOG_LEVEL': 'INFO',
 'AIRFLOW_TASK_REVISION_ID': '4',
 'AIRFLOW_WEB_SERVER_LOGS_ENABLED': 'true',
 'AIRFLOW_WEB_SERVER_LOG_LEVEL': 'WARNING',
 'AIRFLOW_WORKER_LOGS_ENABLED': 'true',
 'AIRFLOW_WORKER_LOG_LEVEL': 'WARNING',
 '_MP_FORK_LOGLEVEL_': '20'}
john-jac commented 2 years ago

https://github.com/aws/aws-mwaa-local-runner/pull/149 allows setting ENV