search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.4k stars 3.97k forks source link

[source-MSSQL] Full refresh Overwrite does not work in some cases #42041

Open coespain opened 1 month ago

coespain commented 1 month ago

Connector Name

source-mssql

Connector Version

4.0.35

What step the error happened?

During the sync

Relevant information

I am doing a sync from an SQL Server to postgres DB.

When upgrading to the latest versions of the airbyte/source-mssql source we have detected that for connections configured with the “Scan Changes with User Defined Cursor” update method for some tables with Sync mode: Full refresh Overwrite, it does not work correctly. It only happens for some tables, we have not been able to determine what is different, apparently these tables are the same as the others. The first execution does it correctly, but from the second synchronization it doesn't read records and empties the destination table

airbyte_sync

Relevant log output

No response

Contribute

akashkulk commented 1 month ago

I think this is related to this issue : https://github.com/airbytehq/oncall/issues/5935. @benmoriceau to confirm.

akashkulk commented 1 month ago

@coespain is the issue occurring between syncs or attempts within the sync?

benmoriceau commented 1 month ago

I am not sure that it is related. Would it be possible to have the platform version running this?

mahadeveaswar commented 1 month ago

@benmoriceau We are also facing similar issue with sync between MSSQL and Redshift. Running Airbyte v0.52.0

[Source] MSSQL Connector: v4.0.21 [Destination] Redshift Connector: v2.6.1

pholterman commented 1 month ago

@benmoriceau We are also facing this issue with sync between MySQL and Snowflake. Airbyte version: 0.50.35 Source: MySQL v3.6.5 Destination: Snowflake v3.10.1

However, it is not that the second time it always has 0 results, sometimes it processes new rows, as if it is an incremental load. image

firewall413 commented 1 week ago

same issue here! Airbyte: 0.57.3 Microsoft SQL Server 4.1.8 Destination: S3