search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.19k stars 4.14k forks source link

[destination-databricks] Create or Replace Table Issue #46988

Open philk1991 opened 1 month ago

philk1991 commented 1 month ago

Connector Name

destination-databricks

Connector Version

3.3

What step the error happened?

During the sync

Relevant information

We're testing the new version of the databricks lakehouse connector out, using HubSpot as the source.

The initial sync works fine but after this we get an error. It looks like the below, trying to create or replace the table that exists? This is an incremental stream so it shouldn't be doing that anyway. Looks like another PR was meant to have fixed this but we're still seeing this issue.

Relevant log output

` Error Code: 0, SQL state: 42P07, Query: CREATE TA***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_ALREADY_EXISTS] org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException: [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view `airbyte_dev`.`companies` because it already exists. Choose a different name, drop the existing object, add the IF NOT EXISTS clause to tolerate pre-existing objects, add the OR REPLACE clause to replace the existing materialized view, or add the OR REFRESH clause to refresh the existing streaming table. SQLSTATE: 42P07 `

Contribute

marcosmarxm commented 2 days ago

I've added the issue to the connector backlog, but it won't be prioritized for now. If this issue is affecting you, please comment or give it a 👍. This helps the team understand the number of users facing the same problem so that high-impact issues can be prioritized.