search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

'template_fields' in 'EmrServerlessDeleteApplicationOperator' should be a tuple, not a string #21

Closed fventuri-availity closed 2 years ago

fventuri-availity commented 2 years ago

It's me again. I noticed that the in the Airflow operators for EMR Serverless you have this line (https://github.com/aws-samples/emr-serverless-samples/blob/main/airflow/emr_serverless/operators/emr.py#L239):

    template_fields: Sequence[str] = "application_id"

The problem with it is that Python converts the string 'application_id' into the list: ['a', 'p', 'p', 'l', ...], and therefore when Airflow/MWAA tries to use that operator it fails with the error message:

AttributeError: 'EmrServerlessDeleteApplicationOperator' object has no attribute 'a'

I found a good explanation in this SO question: https://stackoverflow.com/questions/56845602/python-airflow-error-attributeerror-xsensor-object-has-no-attribute-l

Based on that I changed that line to:

    template_fields: Sequence[str] = ("application_id",)

and after that change, the operator 'EmrServerlessDeleteApplicationOperator' works.

Franco

dacort commented 2 years ago

Thanks, Franco, this was a mistake I made when syncing the code from the Airflow PR. I'll fix it up today.

dacort commented 2 years ago

Thanks again - fixed in v1.0.1.