search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

fixed function typos that prevented seeing job errors #17

Closed sariabod closed 2 years ago

sariabod commented 2 years ago

Issue #, if available:

Description of changes: When a job would fail, instead of getting the EMR failure message code would dump: AttributeError: 'EmrServerlessStartJobOperator' object has no attribute 'job_id'

After fixing that, we get the below: AttributeError: 'EMRServerless' object has no attribute 'describe_job_run'

The two files were updated and now we can see the actual job error: Example: airflow.exceptions.AirflowException: EMR Serverless job failed. Final state is FAILED. job_run_id is xxxxxxxx. Error: Job execution failed, please check complete logs in configured logging destination. ExitCode: 1. Last few exceptions: IndentationError: unexpected indent...

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

dacort commented 2 years ago

Thanks @sariabod for the contribution! Will get this merged and get a new release out.

As an FYI - There is a PR open in the Airflow repository for EMR Serverless support ( https://github.com/apache/airflow/pull/25324 ) that will hopefully be included in the next official provider release. 🙌