search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
150 stars 74 forks source link

Getting "No module named 'airflow.compat'" #23

Closed smunveruddin-mwb closed 2 years ago

smunveruddin-mwb commented 2 years ago

Hi Team,

While trying to test EMRServerless operator using the github link in requirements file, I am getting the following error

"Broken DAG: [/usr/local/airflow/dags/EMR_Serverless.py] Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/emr_serverless/operators/emr.py", line 22, in from emr_serverless.hooks.emr import EmrServerlessHook File "/usr/local/lib/python3.7/site-packages/emr_serverless/hooks/emr.py", line 23, in from airflow.compat.functools import cached_property ModuleNotFoundError: No module named 'airflow.compat' "

Versions used in Docker container:

It would be great if someone can suggest me workarounds.

Thanks

dacort commented 2 years ago

Hi @smunveruddin-mwb - I've only tested this with Airflow 2.2.2. I believe the airflow.compat package was added in 2.2.0 so that could be the reason for the exception here.

Can you upgrade to 2.2.x? Or are you pinned on 2.1.0?

smunveruddin-mwb commented 2 years ago

Hi @dacort, Right now we are not looking at Airflow upgrade. I have made a workaround to add the repo to plugins folder and import cached_property from cached_property package, since it works the same way i believe. The issue has been resolved upon trying this workaround and I am able to submit jobs on EMR Serverless Application.

Thanks for responding to the issue. Really appreciate it.

dacort commented 2 years ago

Sounds good and glad you got it working! For now I'll close this as I don't plan to add support for Airflow <2.2.