search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15k stars 3.87k forks source link

Destination Databricks: Premature Timeout during Databricks Cluster Startup #29050

Open tanawatpan opened 12 months ago

tanawatpan commented 12 months ago

Connector Name

destination-databricks

Connector Version

1.1.0

What step the error happened?

During the sync

Revelant information

OS Version / Instance: ubuntu-jammy-22.04 on AWS EC2 m4.large Memory / Disk: 8Gb/30 GB Deployment: Docker Airbyte Version: 0.50.11 Source Connector version: MySQL (2.1.0) Destination Connector version: Databricks Lakehouse (1.1.0)

When my Databricks Cluster is already up and running, the syncs work perfectly. The problem arises when the sync operation triggers the startup of a stopped Databricks cluster. Airbyte times out before the cluster is fully up and running, resulting in a connection timeout error. Message: HikariPool-1 - Connection is not available, request timed out after 60002ms.

This is particularly odd because I've run the same process with a stopped SQL Warehouse cluster, and it seems to work fine. The issue appears to be specific to the Databricks All Purpose clusters.

Is this behavior expected or limitation of Databricks cluster?

Relevant log output

message: io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60002ms.
io.airbyte.commons.exceptions.ConnectionErrorException: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 60002ms.
    at io.airbyte.integrations.destination.jdbc.AbstractJdbcDestination.attemptTableOperations(AbstractJdbcDestination.java:143)
    at io.airbyte.integrations.destination.jdbc.copy.CopyDestination.performCreateInsertTestOnDestination(CopyDestination.java:100)
    at io.airbyte.integrations.destination.jdbc.copy.CopyDestination.check(CopyDestination.java:72)
    at io.airbyte.integrations.destination.jdbc.copy.SwitchingDestination.check(SwitchingDestination.java:56)
    at io.airbyte.integrations.base.IntegrationRunner.runInternal(IntegrationRunner.java:124)
    at io.airbyte.integrations.base.IntegrationRunner.run(IntegrationRunner.java:99)
    at io.airbyte.integrations.destination.databricks.DatabricksDestination.main(DatabricksDestination.java:23)

Contribute

leo-schick commented 7 months ago

I saw the same issue. I think the timeout should be added as configuration option and should be set to 10 minutes by default for clusters. It is not uncommon that it takes between 4-7 minutes until a cluster is spin up.