Fivetran Sensor shows incorrect sync status

[SouravBhowmikDE]

There is an issue that we have been seeing every 2-3 weeks now across multiple kinds of connectors

The issue is: Sometimes, the sync status from the the Fivetran Sensor is not correctly updated even if the sync status is updated in the UI dashboard. Eg: Let's say when a sync is triggered, the sync status in the UI dashboard would change to scheduled, then to syncing and then to synced. All within a few mins. But the sync status from the sensor would still remain stuck at scheduled. We have a task timeout of 1 hour. So, after one hour the task automatically fails. Then we go look in the UI dashboard and see that everything is normal, there were not errors during the sync fivetran_sync_google_ads_fivetran_wait_for_google_ads_20220701_sync_2023-02-09T07_45_00+00_00_1 (1).log

Can you please let us know why is there discrepancy in the status returned by the APi response via the sensor and the status in the UI dashboard?

I'm attaching logs from our Airflow Task for clarity

[SouravBhowmikDE]

Any updates on this? It happened again today for Salesforce connector. The status was stuck at rescheduled forever until Airflow times out after 1 hour

[2023-02-20, 09:49:19 UTC] {{standard_task_runner.py:82}} INFO - Running: ['airflow', 'tasks', 'run', 'fivetran_sync_bedrock', 'fivetran_wait_for_bedrock_sync', 'scheduled__2023-02-20T07:49:00+00:00', '--job-id', '88561', '--raw', '--subdir', 'DAGS_FOLDER/fivetran/fivetran_ingress.py', '--cfg-path', '/tmp/tmp2x44x5s8'] [2023-02-20, 09:49:19 UTC] {{standard_task_runner.py:83}} INFO - Job 88561: Subtask fivetran_wait_for_bedrock_sync [2023-02-20, 09:49:20 UTC] {{task_command.py:376}} INFO - Running <TaskInstance: fivetran_sync_bedrock.fivetran_wait_for_bedrock_sync scheduled__2023-02-20T07:49:00+00:00 [running]> on host ip-172-22-26-42.ec2.internal [2023-02-20, 09:49:20 UTC] {{taskinstance.py:1590}} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=fivetran_sync_bedrock AIRFLOW_CTX_TASK_ID=fivetran_wait_for_bedrock_sync AIRFLOW_CTX_EXECUTION_DATE=2023-02-20T07:49:00+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=scheduled__2023-02-20T07:49:00+00:00 [2023-02-20, 09:49:20 UTC] {{base.py:71}} INFO - Using connection ID 'data_fivetran' for task execution. [2023-02-20, 09:49:24 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = scheduled [2023-02-20, 09:49:55 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:50:28 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:51:00 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:51:31 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:52:03 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:52:35 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:53:07 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:53:39 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:54:11 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:54:43 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:55:15 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:55:47 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:56:19 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:56:51 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:57:23 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:57:55 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:58:27 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:58:59 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 09:59:31 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:00:03 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:00:35 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:01:07 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:01:39 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:02:11 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:02:43 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:03:15 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:03:47 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:04:19 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:04:51 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:05:23 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:05:55 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:06:27 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:06:59 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:07:31 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:08:03 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:08:35 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:09:07 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:09:39 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:10:11 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:10:44 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:11:15 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:11:47 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:12:19 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:12:51 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:13:23 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:13:55 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:14:27 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:14:59 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:15:31 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:16:03 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:16:35 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:17:07 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:17:39 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:18:11 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:18:43 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:19:15 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:19:47 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:20:19 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:20:51 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:21:23 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:21:55 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:22:27 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = syncing [2023-02-20, 10:22:59 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:23:33 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:24:05 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:24:37 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:25:09 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:25:41 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:26:13 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:26:45 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:27:17 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:27:49 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:28:21 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:28:54 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:29:25 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:29:57 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:30:29 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:31:01 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:31:33 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:32:06 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:32:43 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:33:14 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:33:46 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:34:18 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:34:50 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:35:22 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:35:54 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:36:26 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:36:58 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:37:30 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:38:02 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:38:34 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:39:06 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:39:38 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:40:10 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:40:42 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:41:17 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:41:49 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:42:22 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:42:53 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:43:25 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:43:57 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:44:30 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:45:02 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:45:34 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:46:06 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:46:38 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:47:10 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:47:42 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:48:14 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:48:46 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:49:18 UTC] {{fivetran.py:388}} INFO - Connector "terrace_lurking": sync_state = rescheduled [2023-02-20, 10:49:20 UTC] {{timeout.py:68}} ERROR - Process timed out, PID: 2206 [2023-02-20, 10:49:20 UTC] {{taskinstance.py:1851}} ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/sensors/base.py", line 211, in execute time.sleep(self._get_next_poke_interval(started_at, run_duration, try_number)) File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/timeout.py", line 69, in handle_timeout raise AirflowTaskTimeout(self.error_message) airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 2206 [2023-02-20, 10:49:20 UTC] {{taskinstance.py:1401}} INFO - Marking task as FAILED. dag_id=fivetran_sync_bedrock, task_id=fivetran_wait_for_bedrock_sync, execution_date=20230220T074900, start_date=20230220T094919, end_date=20230220T104920 [2023-02-20, 10:49:20 UTC] {{logging_mixin.py:137}} WARNING - /usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/slack/operators/slack.py:66 DeprecationWarning: Provide token as hook argument deprecated by security reason and will be removed in a future releases. Please specify token inSlack APIconnection. [2023-02-20, 10:49:20 UTC] {{logging_mixin.py:137}} WARNING - /usr/local/airflow/.local/lib/python3.10/site-packages/airflow/providers/slack/operators/slack.py:66 FutureWarning: You have not set parameterslack_conn_id. CurrentlySlack APIconnection id optional but in a future release it will mandatory. [2023-02-20, 10:49:20 UTC] {{standard_task_runner.py:100}} ERROR - Failed to execute job 88561 for task fivetran_wait_for_bedrock_sync (Timeout, PID: 2206; 2206) [2023-02-20, 10:49:20 UTC] {{local_task_job.py:159}} INFO - Task exited with return code 1

[SouravBhowmikDE]

Any updates on this? It happened again today for Google connector and Bing connector. The status was stuck at scheduled forever until Airflow times out after 1 hour

Bing logs INFO - Running <TaskInstance: fivetran_sync_bingads_tcr.fivetran_wait_for_bingads_tcr_sync scheduled__2023-02-24T09:37:00+00:00 [running]> on host ip-172-22-13-175.ec2.internal [2023-02-24, 05:39:10 CST] {{taskinstance.py:1590}} INFO - Exporting the following env vars: AIRFLOW_CTX_DAG_OWNER=airflow AIRFLOW_CTX_DAG_ID=fivetran_sync_bingads_tcr AIRFLOW_CTX_TASK_ID=fivetran_wait_for_bingads_tcr_sync AIRFLOW_CTX_EXECUTION_DATE=2023-02-24T09:37:00+00:00 AIRFLOW_CTX_TRY_NUMBER=1 AIRFLOW_CTX_DAG_RUN_ID=scheduled__2023-02-24T09:37:00+00:00 [2023-02-24, 05:39:10 CST] {{base.py:71}} INFO - Using connection ID 'data_fivetran' for task execution. [2023-02-24, 05:39:11 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:39:41 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:40:12 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:40:42 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:41:12 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:41:43 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:42:13 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:42:43 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:43:14 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:43:44 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:44:14 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:44:47 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:45:17 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:45:47 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:46:18 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:46:48 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:47:18 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:47:49 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:48:19 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:48:49 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:49:19 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:49:50 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:50:20 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:50:50 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:51:21 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:51:51 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:52:22 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:52:52 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:53:22 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:53:52 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:54:23 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:54:53 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:55:23 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:55:54 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:56:24 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:56:54 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:57:25 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:57:55 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:58:25 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:58:56 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:59:26 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 05:59:56 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:00:27 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:00:57 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:01:27 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:01:57 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:02:28 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:02:58 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:03:28 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:03:59 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:04:29 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:04:59 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:05:30 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:06:00 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:06:30 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:07:01 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:07:31 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:08:01 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:08:32 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:09:02 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:09:32 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:10:02 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:10:33 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:11:03 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:11:33 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:12:04 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:12:34 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:13:05 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:13:35 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:14:06 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:14:36 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:15:06 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:15:36 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:16:07 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:16:37 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:17:07 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:17:38 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:18:08 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:18:38 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:19:09 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:19:39 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:20:09 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:20:39 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:21:10 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:21:40 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:22:10 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:22:41 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:23:11 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:23:41 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:24:12 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:24:42 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:25:12 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:25:43 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:26:13 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:26:43 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:27:14 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:27:44 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:28:14 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:28:45 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:29:15 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:29:45 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:30:16 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:30:46 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:31:16 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:31:46 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:32:17 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:32:47 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:33:17 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:33:48 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:34:18 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:34:49 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:35:19 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:35:49 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:36:20 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:36:50 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:37:20 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:37:51 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:38:21 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:38:51 CST] {{fivetran.py:388}} INFO - Connector "clubbing_incurred": sync_state = scheduled [2023-02-24, 06:39:10 CST] {{timeout.py:68}} ERROR - Process timed out, PID: 5759 [2023-02-24, 06:39:10 CST] {{taskinstance.py:1851}} ERROR - Task failed with exception Traceback (most recent call last): File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/sensors/base.py", line 211, in execute time.sleep(self._get_next_poke_interval(started_at, run_duration, try_number)) File "/usr/local/airflow/.local/lib/python3.10/site-packages/airflow/utils/timeout.py", line 69, in handle_timeout raise AirflowTaskTimeout(self.error_message) airflow.exceptions.AirflowTaskTimeout: Timeout, PID: 5759

But Fivetran dashboard says all good during the sync

[johnson-jay-l]

Recommend to follow this doc as a guide and use FivetranOperator + FivetranSensor in the same dag. Use the xcom so that the sensor is looking for the correct sync event that is passed to the xcom by FivetranOperator. https://www.fivetran.com/blog/orchestrating-elt-in-airflow-catching-every-sync-with-xcom

Guessing that the rescheduled problem is because of hitting API throttling on a connector. On the UI, the rescheduled syncs are colored purple on the timeline. I'm not aware of a way to handle for that but am interested to find one.

[SouravBhowmikDE]

I just saw this. Will look into it and close the issue if that serves the purpose. Thanks!

[SouravBhowmikDE]

I implemented this solution and it it's been working fine for all situations except for reschedules. Airflow is still timing out after our time out period of 1 hour whenever the connector is rescheduled. Do we have a solution for reschedules yet? @johnson-jay-l

[SouravBhowmikDE]

Any updates?