search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.31k forks source link

EmrCreateJobFlowOperator should not wait_for_completion if wait_for_completion=false deferrable=true #40966

Open ibzx opened 3 months ago

ibzx commented 3 months ago

Apache Airflow version

2.9.3

If "Other Airflow 2 version" selected, which one?

No response

What happened?

When setting global 

AIRFLOW__OPERATORS__DEFAULT_DEFERRABLE =true
EmrCreateJobFlowOperator(wait_for_completion=false)

The operator should adds a deferral trigger to wait for Jobflow create completion status.

What you think should happen instead?

Since we did not ask to wait for completion, the deferrable argument should do nothing. The deferrable argument should only start a trigger when wait_for_completion=true

How to reproduce

See deferrable code here: https://github.com/apache/airflow/blob/460d2095cda6674dda4860d605573b8f1cec7de5/airflow/providers/amazon/aws/operators/emr.py#L827-L849

Sample code

EmrCreateJobFlowOperator(
        task_id="cluster_m5",
        wait_for_completion=false,
        deferrable=true
)

Operating System

Mac OS 14.5 (23F79)

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 3 months ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Prab-27 commented 3 months ago

hello @eladkal , I am using WSL2 set up can I work on this issue here OS is Mac ? .If yes , I would like to work on this issue as my first contribution.

laksh-krishna-sharma commented 3 months ago

May I work on this issue with a little bit of your guidance?

eladkal commented 3 months ago

cc @vincbeck I think you fixed similar problem for emr serverless?

vincbeck commented 3 months ago

Correct. It seems many operators have the same issue. This is a very good first issue so please feel free to work on it and tag me to review it :) Here are some examples: #41191, #41103, #41059

vincbeck commented 3 months ago

@laksh-krishna-sharma Please look at the example I provided, that should help you to fix this issue

laksh-krishna-sharma commented 3 months ago

Thank you, @vincbeck . I will start working on it immediately. I will reach out if I have any questions.