Currently the jira__issue_field_history model (and it's upstream models) have significant incremental logic in order to handle the high volume of data when it comes to issue field history. However, the logic is built to assume all issue field updates are synced in order and that has been seen to not be the case. There are instances where issue field history records are synced "late" and may not come in for a day or so. This then results in an incorrect end model as the incremental strategy assumes all records are synced in order.
Therefore, we should apply a lookback window within the incremental logic to ensure records synced late are captured in subsequent incremental runs. This will result in an impact to the performance of the models. However, if we do this right, it will be insignificant and more accurate which is the route we would prefer.
Describe alternatives you've considered
Running a full refresh periodically to capture these late arrival records.
Are you interested in contributing this feature?
[ ] Yes.
[ ] Yes, but I will need assistance and will schedule time during your office hours for guidance.
fivetran-reneeli
commented
8 months ago
Is there an existing feature request for this?
Describe the Feature
Currently the
jira__issue_field_historymodel (and it's upstream models) have significant incremental logic in order to handle the high volume of data when it comes to issue field history. However, the logic is built to assume all issue field updates are synced in order and that has been seen to not be the case. There are instances where issue field history records are synced "late" and may not come in for a day or so. This then results in an incorrect end model as the incremental strategy assumes all records are synced in order.Therefore, we should apply a lookback window within the incremental logic to ensure records synced late are captured in subsequent incremental runs. This will result in an impact to the performance of the models. However, if we do this right, it will be insignificant and more accurate which is the route we would prefer.
Describe alternatives you've considered
Running a full refresh periodically to capture these late arrival records.
Are you interested in contributing this feature?
Anything else?
No response