Source name
Source pipedrive, with the deals endpoint
Describe the bug
Some of the records in the deals table are not getting updated. For example, this deal in the screenshot attached was updated once on 20th February and once on 27th. But these changes were not picked up by during the data loads and in our table we see the last update for this deal was on 6th February, refer to screenshot attached.
As a temporary fix we did a full reload of the table after which these updates were loaded. But we still don't know what causes the issue and therefore we might face the same issue again in a few days.
It's not like updates are not reflected at all, but only partially loaded. So some of the deals changed might be correctly loaded to the tables but some of them not.
name of the dlt source causing the problem: pipedrive
name if the dlt resource causing the problem: deals endpoint incremental load
which destination you are loading into: BigQuery
any stack traces and logs you can share: Attached screenshots to understand the issue
To Reproduce
Make changes to some deals in pipedrive and run the pipedrive pipeline to extract the updated records
Expected behavior
When a change happens in pipedrive for any one of the endpoints, the next run of the pipeline should be able to correctly update all rows in the extracted tables
Stack traces and other evidence
Attached the code files for the pipeline and our DAG to invoke the pipeline
pipedrive pipeline.zip
Source name Source
pipedrive, with thedealsendpointDescribe the bug Some of the records in the
dealstable are not getting updated. For example, this deal in the screenshot attached was updated once on 20th February and once on 27th. But these changes were not picked up by during the data loads and in our table we see the last update for this deal was on 6th February, refer to screenshot attached.As a temporary fix we did a full reload of the table after which these updates were loaded. But we still don't know what causes the issue and therefore we might face the same issue again in a few days.
It's not like updates are not reflected at all, but only partially loaded. So some of the deals changed might be correctly loaded to the tables but some of them not.
pipedrivedealsendpoint incremental loadTo Reproduce Make changes to some deals in pipedrive and run the pipedrive pipeline to extract the updated records
Expected behavior When a change happens in pipedrive for any one of the endpoints, the next run of the pipeline should be able to correctly update all rows in the extracted tables
Stack traces and other evidence Attached the code files for the pipeline and our DAG to invoke the pipeline pipedrive pipeline.zip
Running environment
dltVersion 0.4.2