search

Barski-lab / cwl-airflow

Python package to extend Airflow functionality with CWL1.1 support
https://barski-lab.github.io/cwl-airflow
Apache License 2.0
185 stars 32 forks source link

Support for managed Airflow via plugin #60

Closed jeremyadamsfisher closed 3 years ago

jeremyadamsfisher commented 3 years ago

Is your feature request related to a problem? Please describe. I would like to use a managed Airflow cluster like MWAA. Therefore, I cannot swap out the airflow executable. It can, however, take custom plugins.

Describe the solution you'd like Distribute cwl-airflow as a plugin or the equivalent to work on a managed Airflow system

michael-kotliar commented 3 years ago

Hi @jeremyadamsfisher,

Thank you for your feature request. We are working on moving CWL-Airflow to the cloud, so smooth integration with MWAA is on our TODO list.

If you can install custom python packages on MWAA, you don't need to swap the original airflow executable. You just need to install CWL-Airflow package and create your DAGs using CWLDAG class. CWL-Airflow uses the original Airflow and only extends its functionality with the ability to parse and execute CWL workflows. Let me know if you have any questions about installation and configuration of CWL-Airflow.

jeremyadamsfisher commented 3 years ago

For others' reference, it seems one can install custom packages on MWAA: https://docs.aws.amazon.com/mwaa/latest/userguide/best-practices-dependencies.html