search

astronomer / airflow-provider-fivetran-async

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
Apache License 2.0
20 stars 7 forks source link

FivetranAsyncOperator Stuck in Deferred State forever #108

Open Nitindr opened 2 weeks ago

Nitindr commented 2 weeks ago

Hello Team!

We have been using FivetranAsyncOperator from a while and was working fine until recently we upgraded the Airflow Version to 2.9.2, since then some of the tasks in our Fivetran DAG gets stuck in Deferred state and never returns the sync completed status even though the sync has completed at the Fivetran end and the task keeps on running until we mark at as success manually.

Airflow Version : V2.9.2 airflow-provider-fivetran-async==2.0.2

Task File

fivetran_chargebee_cp_sync = FivetranOperator( task_id="fivetran_chargebee_cp_sync", fivetran_conn_id="fivetran_conn", connector_id=Variable.get("fivetran_chargebee_cp"), wait_for_completion=True, dag=dag, )

Task Logs

ip-10-100-105-115.ec2.internal
*** Reading remote log from Cloudwatch log_group: airflow-datateam-airflow-dev-Task log_stream: dag_id=Fivetran_All_Connectors_Sync/run_id=scheduled__2024-09-07T12_00_00+00_00/task_id=fivetran_chargebee_cp_sync/attempt=1.log.
[2024-09-08, 05:43:18 IST] {local_task_job_runner.py:120} ▶ Pre task execution logs
[2024-09-08, 05:43:19 IST] {base.py:84} INFO - Using connection ID 'fivetran_conn' for task execution.
[2024-09-08, 05:43:19 IST] {hooks.py:320} INFO - Connector type: chargebee_product_catalog_1, connector schema: chargebee_cp
[2024-09-08, 05:43:19 IST] {hooks.py:321} INFO - Connectors logs at https://fivetran.com/dashboard/connectors/chargebee_product_catalog_1/chargebee_cp/logs
[2024-09-08, 05:43:20 IST] {hooks.py:487} INFO - Connector guild_su*****: sync_state = scheduled
[2024-09-08, 05:43:20 IST] {taskinstance.py:441} ▼ Post task execution logs
[2024-09-08, 05:43:20 IST] {taskinstance.py:2489} INFO - Pausing task as DEFERRED. dag_id=Fivetran_All_Connectors_Sync, task_id=fivetran_chargebee_cp_sync, run_id=scheduled__2024-09-07T12:00:00+00:00, execution_date=20240907T120000, start_date=20240908T001318
[2024-09-08, 05:43:21 IST] {local_task_job_runner.py:237} INFO - Task exited with return code 100 (task deferral)
[2024-09-08, 05:43:21 IST] {local_task_job_runner.py:222} ▲▲▲ Log group end
pankajastro commented 5 days ago

Hey @Nitindr, Do you know what are delta between working and not working?

pankajastro commented 4 days ago

Could you please validate https://github.com/astronomer/airflow-provider-fivetran-async/issues/109#issuecomment-2368036922 and this true then you should report issue in Airflow repo