search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.15k stars 4.13k forks source link

[source-MySQL] incremental append behavies different on first run than other runs #43327

Open pedrohsroque opened 3 months ago

pedrohsroque commented 3 months ago

Connector Name

source-mysql

Connector Version

3.4.10

What step the error happened?

Other

Relevant information

We are replication data from a MySQL database in another timezone using incremental append dedup mode to Snowflake. On the first run (which pulls the entire data) we observed that the timezone for the datetime columns is the same as the original database, but on subsequent runs, it became GMT-0, so now we have part of the data in one timezone and another part in GMT-0, and over time new records are GMT-0, updated records GMT-0 and remaining records from first run are in the original timezone.

When running full refresh, we observed the original timezone for all runs.

image

I found similar issues, but all were closed with no apparent solution.

Relevant log output

No response

Contribute

mekedron commented 3 weeks ago

The difference also with the numeric columns as it's mentioned in this issue: https://github.com/airbytehq/airbyte/issues/38470#issuecomment-2421964712