search

awslabs / autonomous-driving-data-framework

ADDF is a collection of modules, deployed using the SeedFarmer orchestration tool. ADDF modules enable users to quickly bootstrap environments for the process and analysis of autonomous driving data.
Apache License 2.0
109 stars 45 forks source link

[BUG] MWAA version conflict #128

Closed dgraeber closed 1 year ago

dgraeber commented 1 year ago

Describe the bug When deploying core/mwaa, the dags for analysis/rosbag-image fail to load due to references to BatchCreateComputeEnvironmentOperator in dependency libraries.

Triaging this issue is related to the installation of the library amazon-airflow[amazon] in modules/core/mwaa/requirements/requirements.txt. A temporary fix was to remove this reference, but this breaks the dags located in modules/examples/example-spark-dags/example_spark_dags/citibike_all_dag.py

On main and on release/1.1.0 in order to support EMR-on-EKS from MWAA and have back-support for rosbag pipelines, we will need to parametrize the requirements file and allow the manifests to pass in which requirements file is needed for individual use cases (ex. one MWAA module manifest entry for rosbag-image, one MWAA module manifest entry for emr-on-eks).

This change needs to occur to support multiple versions of DAGs.

dgraeber commented 1 year ago

This is the supported list of dependencies for MWAA latest release: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.2/constraints-3.7.txt