search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

Now the EMRServerlessStartJobOperator has the countdown and check_int… #38

Open wesolowskim opened 2 years ago

wesolowskim commented 2 years ago

Now the EMRServerlessStartJobOperator has the countdown and check_interval_seconds parameters.

Issue #, if available: https://github.com/aws-samples/emr-serverless-samples/issues/27

Description of changes: When countdown and check_interval_seconds parameters are set, the hook waits the requested number of seconds for the job copmletion.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

dacort commented 2 years ago

Thanks @wesolowskim! At first glance this looks good. I'll try to do a proper review asap and get an updated release out.

donpayne commented 2 years ago

This would be great to have available! Please make it happen.

dacort commented 2 years ago

We did just have a PR opened in the official Airflow repo for the same: https://github.com/apache/airflow/pull/27784

I'd prefer to use that one as I want to keep this in sync with the Airflow Provider.