search

aws-samples / emr-serverless-samples

Example code for running Spark and Hive jobs on EMR Serverless.
https://aws.amazon.com/emr/serverless/
MIT No Attribution
155 stars 78 forks source link

Parameter countdown cannot be passed to EmrServerlessStartJobOperator #27

Closed EzuA closed 1 year ago

EzuA commented 2 years ago

Hello, I am trying to increase the default parameter countdown in the EmrServerlessStartJobOperator but I cannot do that because the waiter is not receiving it:

            self.hook.waiter(
                get_state_callable=self.hook.conn.get_job_run,
                get_state_args={
                    "applicationId": self.application_id,
                    "jobRunId": response["jobRunId"],
                },
                parse_response=["jobRun", "state"],
                desired_state=EmrServerlessJobSensor.SUCCESS_STATES,
                failure_states=EmrServerlessJobSensor.FAILURE_STATES,
                object_type="job",
                action="run",
            )
dacort commented 2 years ago

@EzuA Thanks for opening this issue - we definitely want to add the ability to configure countdown, hopefully will get to it soon!

liamoneill commented 1 year ago

Is the plan to sync the changes from the now merged upstream PR https://github.com/apache/airflow/pull/27784 into this repo?

My company is using MWAA and unfortunately won't be able to pull in the next release of apache-airflow-providers-amazon because >= 6.1 requires Airflow 2.3.

dacort commented 1 year ago

@liamoneill Yes, that's the plan - I've got an update in progress, just need to push it up soon and cut a new release. :)

Thanks for the feedback re: the 6.1 provider release. I wasn't sure if I would need to continue to maintain this as an independent plugin, but looks like I may need to.

dacort commented 1 year ago

PR is open here, will look to validate and merge next week. :) https://github.com/aws-samples/emr-serverless-samples/pull/42