search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.3k stars 4.15k forks source link

Fetch schema timeout issue with NetSuite connector affecting all destinations without specifying object types #25223

Open luancaarvalho opened 1 year ago

luancaarvalho commented 1 year ago

Description

I am experiencing an issue with the Airbyte NetSuite connector while trying to fetch the schema without specifying any object types in Airbyte v0.45.8, which is the latest version available via the Helm package, as well as in previous versions. It appears that the connector does not wait long enough for the NetSuite metadata-catalog API to respond, causing the fetch schema operation to fail. The API's response time is around 98.437 seconds. This issue occurs when connecting the NetSuite source connector to any destination connector (e.g., e2e, S3, etc.).

Steps to reproduce the issue

  1. Install Airbyte v0.45.8 or earlier using the Helm package.
  2. Set up the NetSuite connector in Airbyte with the required credentials and configurations.
  3. Connect the NetSuite source connector to any destination connector (e.g., e2e, S3, etc.) without specifying any object types.
  4. Attempt to fetch the schema for the configured NetSuite objects.
  5. Observe that the fetch schema operation times out before receiving a response from the NetSuite metadata-catalog API, which takes around 98.437 seconds to respond.

Expected behavior The Airbyte NetSuite connector should wait for a sufficient amount of time for the NetSuite metadata-catalog API to respond, allowing the fetch schema operation to complete successfully and enabling the proper configuration of the data pipeline between the NetSuite source and destination connectors.

Additional context This issue is causing difficulties in setting up and configuring the NetSuite connector in Airbyte, as the schema information is essential for proper data extraction and syncing with destination connectors, even without specifying any object types. The problem persists across multiple versions of Airbyte, including the latest version available via the Helm package (v0.45.8) and earlier versions. Follow the Images showing the response time in NetSuite and how airbyte looks after the complete response in NetSuite. image (6) image (7)

I kindly request the Airbyte maintainers to investigate this issue and provide a solution that allows sufficient time for the NetSuite metadata-catalog API to respond during the fetch schema operation, ensuring seamless integration between the NetSuite source connector and various destination connectors without specifying object types, in both the latest version and previous versions of Airbyte.

Thank you for your attention to this matter.

d-shames commented 1 year ago

I am having the exact issue - thanks for outlining in detail @luancaarvalho

jhuettner commented 1 year ago

Same thing here, Netsuite is sloooooow and times out. Not really the connectors fault, NetSuite API is just slow, but connector needs to accomodate the delay please.

lane-certn commented 9 months ago

Also consistently hitting this issue.

d-shames commented 9 months ago

fwiw i found a workaround by specifying the objects I need when setting up the source

lane-certn commented 9 months ago

fwiw i found a workaround by specifying the objects I need when setting up the source

This worked immediately for us as well. Thank you!

kamlesh-zageno commented 1 month ago

Using object type I can find the tables (account, customer, employee, vendor etc) but couldn't find the transaction and deleted. Does anyone know how to solve the issue?

d-shames commented 1 month ago

Using object type I can find the tables (account, customer, employee, vendor etc) but couldn't find the transaction and deleted. Does anyone know how to solve the issue?

the transaction table does not exist as you'd normally see it. you have to pull individual transaction types, ie. invoice, creditmemo, journalentry, etc. The deletes are also not accessible, our team is pulling them manually from the UI every month and adding them to a dbt seed file.

kamlesh-zageno commented 1 month ago

Using object type I can find the tables (account, customer, employee, vendor etc) but couldn't find the transaction and deleted. Does anyone know how to solve the issue?

the transaction table does not exist as you'd normally see it. you have to pull individual transaction types, ie. invoice, creditmemo, journalentry, etc. The deletes are also not accessible, our team is pulling them manually from the UI every month and adding them to a dbt seed file.

Thanks @d-shames 👍 Is there any documentation that I can refer to? What are the objects falls under Transaction

d-shames commented 1 month ago

Using object type I can find the tables (account, customer, employee, vendor etc) but couldn't find the transaction and deleted. Does anyone know how to solve the issue?

the transaction table does not exist as you'd normally see it. you have to pull individual transaction types, ie. invoice, creditmemo, journalentry, etc. The deletes are also not accessible, our team is pulling them manually from the UI every month and adding them to a dbt seed file.

Thanks @d-shames 👍 Is there any documentation that I can refer to? What are the objects falls under Transaction

I believe you can pull anything listed here (don't believe the airbyte connector is case senstive when defining objects)