[source-salesforce] Certain Records Not Syncing Despite Matching Cursor Date Range

ahmedhamidibr commented 1 month ago

Connector Name

Salesforce

Connector Version

2.5.23

What step the error happened?

Other

Relevant information

I am experiencing an issue where certain records from Salesforce are not being synced to Snowflake despite having the same createddate as other records that are successfully synced. I am using version 2.5.23 of the Salesforce connector.

Details:

I have identified that the records with the following Id and createddate are not being synced:

Id	In Snowflake	createddate
017bW00000HKErBQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKErCQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKErDQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKErGQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKErHQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKErNQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKErOQAX	FALSE	2024-07-24T14:05:58.000+0000
017bW00000HKJZlQAP	FALSE	2024-07-24T14:06:41.000+0000
017bW00000HKJZmQAP	FALSE	2024-07-24T14:06:41.000+0000
017bW00000HKJZsQAP	FALSE	2024-07-24T14:06:41.000+0000
017bW00000HKJZtQAP	FALSE	2024-07-24T14:06:41.000+0000
017bW00000HKO12QAH	FALSE	2024-07-24T14:51:51.000+0000
017bW00000HLS0nQAH	FALSE	2024-07-24T19:00:58.000+0000
017bW00000HLS0oQAH	FALSE	2024-07-24T19:00:58.000+0000

However, other records with different Id but same createddate date are synced successfully:

Id	In Snowflake	createddate
017bW00000HG84WQAT	TRUE	2024-07-24 06:30:25.000 +0000
017bW00000HG84bQAD	TRUE	2024-07-24 06:30:25.000 +0000
017bW00000HG84TQAT	TRUE	2024-07-24 06:30:25.000 +0000
017bW00000HG84cQAD	TRUE	2024-07-24 06:30:25.000 +0000
017bW00000HG84VQAT	TRUE	2024-07-24 06:30:25.000 +0000

The connection state for the Airbyte connection to this object is as follows:

{
  "streamState": {
    "slices": [
      {
        "end": "2024-07-24T09:39:33.634Z",
        "start": "2021-03-10T00:00:00.000Z"
      }
    ],
    "state_type": "date-range"
  }

Clearly, the unsynced records fall within the specified date range.

Can any one suggest a solution? If this issue persists, despite the fixes mentioned for version 2.1.X, it would be helpful to have a resolution or a workaround.

NB: This is just one example I have this issue for 9 more other objects

Relevant log output

No response

Contribute

[ ] Yes, I want to contribute

FredericoCoelhoNunes commented 1 week ago

Hi @ahmedhamidibr , I am also experiencing some issues with missing records (in my case when writing to s3), as detailed in this issue.

Whenever you do a full refresh, are your records successfully synced? Or not at all?

ahmedhamidibr commented 1 week ago

Hi @ahmedhamidibr , I am also experiencing some issues with missing records (in my case when writing to s3), as detailed in this issue.

Whenever you do a full refresh, are your records successfully synced? Or not at all?

Hi @FredericoCoelhoNunes , yes when I do a full refresh its usually successfully synced.

airbytehq / airbyte