search

aws / amazon-mwaa-docker-images

Apache License 2.0
24 stars 11 forks source link

MWAA CLI - open more supported on airflow commands #148

Open wilsonhooi86 opened 4 days ago

wilsonhooi86 commented 4 days ago

Overview

Currently MWAA has limited supported airflow commands. As listed here: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html

We have a few scenarios that requires some unsupported MWAA CLI command to work. Example:

  1. Airflow pool import Command: airflow pools import [-h] [-v] FILEPATH Link: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#pools Issue: This command is not supported in MWAA as it returns Response [500] error. Reason needed: This will help us to import a list of airflow pools at one go using json file. Currently we have to use airflow pools setcommand to insert/update and airflow pools delete command to delete one airflow pool at a time. This takes alot of time when we have 200++ airflow pools to insert/update/delete daily. This may cause performance issue to Airflow metadata with many airflow CLI command calls each time.

  2. Airflow create / delete role Command: airflow roles create [-h] [-v] [role [role ...]] or airflow roles delete [-h] [-v] [role [role ...]] Link: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#roles Issue: This command is not supported in MWAA as it returns Response [500] error. Reason needed: To dynamically create new roles programmatically, so that we can auto assign specified roles to users who access specified dags.

  3. Airflow assign role to user Command: airflow users add-role [-h] [-e EMAIL] -r ROLE [-u USERNAME] [-v] Link: https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#users Issue: This command is not supported in MWAA as it returns Response [500] error. Reason needed: Based on point no.2 , after the roles are programmatically created, we would like to programmatically assign the respective airflow custom roles to the users.

Acceptance Criteria

Able to support all the airflow CLI commands for above scenarios.

Additional Info

Unlocking the unsupported Airflow CLI commands in MWAA will greatly help us in many scenarios such as custom role creation and airflow pool assignments. There are many more scenarios we wish to use Airflow CLI for automation feature in future as well.

Kytha commented 1 day ago

Hi Wilson, I'll look into this. For # 2, airflow roles create is listed as a supported command for v2.6+, but I will see if I can recreate your issue. For the other 2 I'll need to see what (if any) blockers there are to supporting these commands.

wilsonhooi86 commented 23 hours ago

Good Day @Kytha , thank you for looking into this issue. I tried the airflow roles create command recently but still get Response [500] error.

image

FYI, I used the sample code from this page to run other mwaa cli command: https://docs.aws.amazon.com/mwaa/latest/userguide/airflow-cli-command-reference.html#airflow-cli-commands-tutorials

Also to add on, may I ask is there why MWAA blocks selected airflow CLI commands? It would be better to unlock all the airflow CLI commands in MWAA, so that we can dynamically programmatically update the MWAA from backend for various scenarios like airflow pools import list and custom roles feature (Needed commands to work: airflow roles add/delete, airflow users create/add-role/delete/remove-role)