search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.07k stars 4.11k forks source link

salesforce records missing with old lastmodified/created date, potential slicer/filter issue #37528

Closed wwfch-cyrill closed 6 months ago

wwfch-cyrill commented 6 months ago

Connector Name

source-salesforce

Connector Version

2.5.6

What step the error happened?

During the sync

Relevant information

This issue happens when adding a new stream entry for a salesforce object. The object has records in salesforce already. These records are often older than 2 years (createdDate).

Sync settings: Incremental | Append + Deduped

Steps:

  1. Choose object, e.g. PricebookEntry
  2. Add new connection (salesforce to snowflake)
  3. Sync
  4. Compare records: Lots of old records are not synced

Workarounds not working: Resetting the stream, changing the date of the state (SystemModstamp) to a date older than the oldest record in salesforce.

Workaround working: update the records in salesforce so that their lastModifiedDate modifies to today. Then airbyte is picking up the reocrds.

From the data it looks like airbyte is not querying data older than 2 years, also this is what the state file says after initial sync:

"streamDescriptor": {
  "name": "Member__c"
},
"streamState": {
  "slices": [
    {
      "end": "2024-04-24T06:17:01.691Z",
      "start": "2022-04-24T06:16:59.000Z"
    }
  ],
a433X00000255fCQAQ  24.02.2021 10:45    12.08.2021 13:01    missing
a433X000002WasBQAS  08.04.2021 16:54    12.08.2021 13:01    missing
a433X000004lkqHQAQ  01.07.2021 09:14    12.08.2021 13:01    missing
a433X000002Wz91QAC  15.04.2021 13:40    21.09.2021 11:10    missing
a433X000000AWGQQA4  16.09.2021 13:07    10.12.2021 11:09    missing
a433X000000Ad7vQAC  31.01.2022 09:40    31.01.2022 09:40    missing
a433X000000Ad9wQAC  31.01.2022 15:55    31.01.2022 16:02    missing
a433X000000AdBiQAK  01.02.2022 15:40    03.02.2022 10:26    missing
a433X000000AdUAQA0  03.02.2022 14:20    03.02.2022 14:22    missing
a433X000000AdTCQA0  03.02.2022 09:59    03.02.2022 15:39    missing
a433X000000AdjDQAS  10.02.2022 17:39    11.02.2022 17:45    missing
a433X000000AdyTQAS  19.02.2022 13:45    19.02.2022 13:48    missing
a433X000000Af83QAC  26.03.2022 19:04    26.03.2022 19:27    missing
a433X000000Af88QAC  26.03.2022 19:21    26.03.2022 19:27    missing
a433X000000Af89QAC  26.03.2022 19:27    26.03.2022 19:29    missing
a433X000000Af8DQAS  26.03.2022 19:32    26.03.2022 19:33    missing
a433X000000Af8IQAS  26.03.2022 19:35    26.03.2022 19:36    missing
a433X000000Af8NQAS  26.03.2022 19:38    26.03.2022 19:39    missing
a433X000000AdBOQA0  01.02.2022 14:57    31.03.2022 12:08    missing
a433X000000AfKsQAK  01.04.2022 16:08    01.04.2022 16:08    missing
a433X000000AfMjQAK  04.04.2022 11:32    04.04.2022 11:34    missing
a433X000000AfPiQAK  05.04.2022 14:31    07.04.2022 14:52    missing
a433X000000AfOBQA0  04.04.2022 16:20    07.04.2022 15:31    missing
a433X000000AfPYQA0  05.04.2022 14:19    14.04.2022 16:51    missing
a433X000000AfPZQA0  05.04.2022 14:28    21.04.2022 08:42    missing
a433X000000AftFQAS  22.04.2022 13:18    22.04.2022 13:20    missing
a433X000000AfNmQAK  04.04.2022 15:22    25.04.2022 14:28    a433X000000AfNmQAK
a433X000000AfNwQAK  04.04.2022 15:30    25.04.2022 14:56    a433X000000AfNwQAK
a433X000000AfNhQAK  04.04.2022 15:19    25.04.2022 14:56    a433X000000AfNhQAK
a433X000000AfNrQAK  04.04.2022 15:28    25.04.2022 14:57    a433X000000AfNrQAK
a433X000000AfO1QAK  04.04.2022 15:58    25.04.2022 14:58    a433X000000AfO1QAK
a433X000000Afr9QAC  21.04.2022 17:12    25.04.2022 15:22    a433X000000Afr9QAC
a433X000000AfpSQAS  20.04.2022 21:08    26.04.2022 08:23    a433X000000AfpSQAS
a433X000000AgS2QAK  12.05.2022 15:13    12.05.2022 15:51    a433X000000AgS2QAK
a433X000000Ad4cQAC  27.01.2022 15:43    23.05.2022 16:48    a433X000000Ad4cQAC
a433X000000Ad4hQAC  27.01.2022 15:46    23.05.2022 16:55    a433X000000Ad4hQAC
a433X000000AfpwQAC  21.04.2022 11:31    24.05.2022 10:03    a433X000000AfpwQAC
a433X000000Ag3vQAC  28.04.2022 13:58    07.06.2022 09:38    a433X000000Ag3vQAC
a433X000000Ag4AQAS  28.04.2022 14:20    07.06.2022 09:39    a433X000000Ag4AQAS
a433X000000AhXmQAK  13.06.2022 09:52    13.06.2022 10:12    a433X000000AhXmQAK
a433X000000AhaHQAS  13.06.2022 14:42    13.06.2022 14:50    a433X000000AhaHQAS
a433X000000AgRYQA0  12.05.2022 13:01    14.06.2022 16:57    a433X000000AgRYQA0
a433X000000AdiAQAS  10.02.2022 15:57    30.06.2022 09:49    a433X000000AdiAQAS
a433X000000AgEZQA0  05.05.2022 09:30    30.06.2022 09:50    a433X000000AgEZQA0

Relevant log output

No response

Contribute

wwfch-cyrill commented 6 months ago

Just saw the config option:

image

Closing this issue now.