search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.51k stars 3.99k forks source link

[source-netsuite] long running historical syncs eventually restart old slices #35745

Open maximum-pixels opened 6 months ago

maximum-pixels commented 6 months ago

Connector Name

source-netsuite

Connector Version

0.1.3

What step the error happened?

During the sync

Relevant information

During longer running syncs (e.g. initial syncs with a few 100k records) the connector eventually starts re-extracting old records and never ends up finishing the syncs. I've added an upper cursor end parameter to the source so that historic extraction can be done in stages so that a sync can finish, the end can be extended, and the following sync will only grab new data.

Relevant log output

No response

Contribute

marcosmarxm commented 6 months ago

@maximum-pixels can you update the title to have a more meaningful description?

maximum-pixels commented 6 months ago

sorry @marcosmarxm I mainly made it for the patch to the connector I submitted here in this PR

marcosmarxm commented 5 months ago

Thanks @maximum-pixels sorry the delay to get your contribution merged. There is quite a large community contribution backlog, hope to get soon to yours.