search

astronomer / airflow-provider-fivetran-async

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran
Apache License 2.0
20 stars 7 forks source link

Fivetran Sync Issue when running Airflow DAG #106

Open prashanttiwaritrellix opened 2 months ago

prashanttiwaritrellix commented 2 months ago

Deferred Sync Log.log successful Sync Log.log

Hi Team,

I am running Fivetran Sync using Airflow Astronomer by importing "from fivetran_provider_async.operators import FivetranOperatorAsync" , when job is running in Airflow then sometime Fivetran completes the job but does not send signal to Airflow, so my airflow job get stuck in deferred mode for infinite time. Please help me how to fix this.

Airflow version - Astro Runtime 8.8.0 (based on Airflow v)

We are using Airflow-provider-fivetran-async package version - 1.0.0a5 (also tried 2.1.0)

pankajastro commented 2 months ago

Could you please upgrade the package and test with https://github.com/astronomer/airflow-provider-fivetran-async/releases/tag/2.1.0

pankajastro commented 2 months ago

I noticed you've mentioned trying version 2.1.0 as well, but FivetranOperatorAsync isn't available in any version beyond 2.0.0. A similar issue https://github.com/astronomer/airflow-provider-fivetran-async/issues/93 was reported earlier. However, after upgrading, that issue was resolved for them.

prashanttiwaritrellix commented 2 months ago

Hi,

I am using FivetranOperator in place of FivetranOperatorAsync with this version and i have tried upgrading astro version but issue still persists. Please advise.

Regards, Prashant Tiwari

On Mon, 2 Sep 2024 at 9:19 PM, Pankaj Singh @.***> wrote:

I noticed you've mentioned trying version 2.1.0 as well, but FivetranOperatorAsync isn't available in any version beyond 2.0.0. A similar issue #93 https://github.com/astronomer/airflow-provider-fivetran-async/issues/93 was reported earlier. However, after upgrading, that issue was resolved for them.

— Reply to this email directly, view it on GitHub https://github.com/astronomer/airflow-provider-fivetran-async/issues/106#issuecomment-2325017177, or unsubscribe https://github.com/notifications/unsubscribe-auth/BB733JXHQWIYETSGLYN2OD3ZUSCG5AVCNFSM6AAAAABNP7Z4HOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRVGAYTOMJXG4 . You are receiving this because you authored the thread.Message ID: @.*** com>

pankajastro commented 2 months ago

Hey, I'm curious whether you’ve recently started using this package or if there have been any changes or upgrade on your end that might have led to this error. Could you provide the steps and DAG that would help in reproducing the issue?

There little chance why it would work for some run and not work some run give that Fivetran response is correct and Airflow behaving correctly.

I think it might be helpful to create a support ticket with Astronomer. They could have better access to the system and might be able to assist you more effectively.

prashanttiwaritrellix commented 2 months ago

Hi Pankaj,

No, we have not done any upgrade or change. I have already raised a ticket with Astro also and they are suspecting some API configuration issue from Fivetran end. If possible please check for any issue. I have attached the complete mail chain where I had a number of interactions with the Astro support team.

DAG Name - TF_Asset_LineItem_updates Regards, Prashant Tiwari Call:- 7353465737 *Mail - @. @.>*

On Wed, Sep 4, 2024 at 11:56 PM Pankaj Singh @.***> wrote:

Hey, I'm curious whether you’ve recently started using this package or if there have been any changes or upgrade on your end that might have led to this error. Could you provide the steps and DAG that would help in reproducing the issue?

There little chance why it would work for some run and not work some run give that Fivetran response is correct and Airflow behaving correctly.

I think it might be helpful to create a support ticket with Astronomer. They could have better access to the system and might be able to assist you more effectively.

— Reply to this email directly, view it on GitHub https://github.com/astronomer/airflow-provider-fivetran-async/issues/106#issuecomment-2329715564, or unsubscribe https://github.com/notifications/unsubscribe-auth/BB733JRQ3AVZHJE4IYDXI7LZU5GFRAVCNFSM6AAAAABNP7Z4HOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRZG4YTKNJWGQ . You are receiving this because you authored the thread.Message ID: @.*** com>

pankajastro commented 1 month ago

Hey @prashanttiwaritrellix, Do you know what are delta between working and not working?

pankajastro commented 1 month ago

Could you please validate https://github.com/astronomer/airflow-provider-fivetran-async/issues/109#issuecomment-2368036922 and this true then you should report issue in Airflow repo