search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.04k stars 4.11k forks source link

[source-jira] `issue_worklog` Overwrite Sync not uploading all data #42951

Open nachosan2 opened 3 months ago

nachosan2 commented 3 months ago

Connector Name

source-jira

Connector Version

3.0.11

What step the error happened?

Updating the connector

Relevant information

We updated the jira connector from version 1.2.0 to version 3.0.11 and we are having a problem with the issue_worklog stream. We have it in a full refresh ovewrite mode. When we resetted the connection, it worked fine, but whenever it syncronizes by the cron we set, its only bringing few results, and not the whole worklogs. I tried resetting only the stream and it worked again, but the later syncronization is not working ok. I also made a manual syncronization and it happened the same with a few records updated.

I didn't find any error in the logs. Im attaching the last sync made with the cron that brought only 120 records to the issue_worklogs table

b2e2d5d9_4956_40f8_944c_69c8d4a32143_logs_5959_txt.txt

Relevant log output

No response

Contribute

marcosmarxm commented 2 months ago

How many records are you expecting instead of the 120? Are you able to run a manual request and ensure latest Jira API return the proper data?

nachosan2 commented 2 months ago

If I run a manual request I get the same result, just a few (I can't find a pattern because it changes the number of records each time).

The way I found out that it works fine is if my reset the stream, like this:

In that case is the way Im getting the whole results, that right now Im getting around 6700 worklogs in issue_worklogs