search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.07k stars 4.11k forks source link

Salesforce <-> Airbyte full sync blocked to 14900 records for OpportunityLineItems #19014

Closed louisdauvois closed 1 year ago

louisdauvois commented 1 year ago

Is this your first time deploying Airbyte?: No OS Version / Instance: MacOS Memory / Disk: 500GO Deployment: Docker Airbyte Version: What version are you using now? Source name/version: 0.40.18 Destination name/version: Step: Create a new connection between Salesforce and CSV, run a full sync Description: When we try to do a full sync on our Salesforce Objects the sync stop at 14900 records but when we do in in salesforce or via api we get 49000+ records.

Seems to have the problem only on OpportunityLineItems since we get approx good number for the account object.

birdieorbust commented 1 year ago

I am seeing the same issue when doing a full sync on the Account object and not all records being returned.

arsenlosenko commented 1 year ago

fix for this issue will be in scope of this PR: https://github.com/airbytehq/airbyte/pull/24888

roman-yermilov-gl commented 1 year ago

@louisdauvois Could you provide us with any logs so we can investigate the process?

roman-yermilov-gl commented 1 year ago

My recommendation would be to use incremental sync as it has checkpointing. Actually full-refresh sync is also using smaller batches now so it worth to try it again. And it also important to check start date because if it's not set, then sync will be for last 2 years