search

apache / airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
https://airflow.apache.org/
Apache License 2.0
37.12k stars 14.31k forks source link

FivetranOperator stuck in deferred state till it times out #42465

Closed dgrewal-tc closed 2 weeks ago

dgrewal-tc commented 1 month ago

Apache Airflow version

Other Airflow 2 version (please specify below)

If "Other Airflow 2 version" selected, which one?

2.9.2

What happened?

This is in relation to https://github.com/astronomer/airflow-provider-fivetran-async/issues/109

As mentioned in the comments, When we upgraded Airflow from 2.8.1 to 2.9.2, FivetranOperator in airflow-provider-fivetran-async package started giving issues abruptly.

We are using the FivetranOperator to trigger a Fivetran connector. The connector completes in 5-10 minutes, but the Airflow task does not finish. It remains in a deferred state until it times out and fails.

Airflow Version : 2.9.2 airflow-provider-fivetran-async==2.1.0

Failed logs

2024-09-15, 10:26:53 UTC] INFO - Pausing task as DEFERRED.
[2024-09-15, 10:26:54 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 11:26:58 UTC] INFO - Resuming after deferral
[2024-09-15, 11:26:59 UTC] INFO - Executing <Task(FivetranOperator)> on 2024-09-14 08:00:00+00:00
[2024-09-15, 11:26:59 UTC] INFO - Started process 9527 to run task
[2024-09-15, 11:26:59 UTC] INFO - Job 918648: Subtask
[2024-09-15, 11:26:59 UTC] ERROR - Task failed with exception
Traceback (most recent call last):
airflow.exceptions.AirflowTaskTimeout

Success logs

[2024-09-15, 12:12:51 UTC] INFO - Pausing task as DEFERRED. 
[2024-09-15, 12:12:52 UTC] INFO - Task exited with return code 100 (task deferral)

[2024-09-15, 12:12:52 UTC] INFO - Using connection ID 'id' for task execution.
[2024-09-15, 12:12:52 UTC] INFO - sync is still running...
[2024-09-15, 12:12:52 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:14:53 UTC] INFO - sync is still running...
[2024-09-15, 12:14:53 UTC] INFO - sleeping for 120 seconds.
[2024-09-15, 12:16:53 UTC] INFO - sync is still running...
[2024-09-15, 12:16:53 UTC] INFO - sleeping for 120 seconds.

[2024-09-15, 12:19:00 UTC] INFO - Resuming after deferral
[2024-09-15, 12:19:00 UTC] INFO - Executing <Task(FivetranOperator):> on 2024-09-14 08:00:00+00:00
[2024-09-15, 12:19:00 UTC] INFO - Started process 11601 to run task
[2024-09-15, 12:19:00 UTC] INFO - Job 918720: Subtask
[2024-09-15, 12:19:00 UTC] INFO - Fivetran connector finished syncing at 2024-09-15 12:17:07.840000+00:00
[2024-09-15, 12:19:00 UTC] INFO - Marking task as SUCCESS.
[2024-09-15, 12:19:00 UTC] INFO - Task exited with return code 0

Can someone please have a look ?

Thanks

What you think should happen instead?

No response

How to reproduce

Use FivetranOperator. This issue does not happen every time we run it. It is very random.

Operating System

Amazon Linux. VERSION=2023

Versions of Apache Airflow Providers

No response

Deployment

Amazon (AWS) MWAA

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

Code of Conduct

boring-cyborg[bot] commented 1 month ago

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.