[Bug] Incremental runs cause null daily_history fields when issue is updated after being Closed for 30 days

kenzie-marsh commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues

Describe the issue

When issues have had Status = Closed for 30+ days and are then updated, all columns other than date_day, issue_id, issue_day_id on the jira__daily_issue_field_history table become null. Since the most recent date for the issue receives all null values, the jira__issue_enhanced table is updated to all null values for that issue as well. Full-refresh-ing the package fixes this problem by creating a row for every date_day in between the updates.

Relevant error log or model output

Screenshot of problematic output included below.

Expected behavior

The daily table should record the new/current values of all fields on an issue regardless of how long it has been Closed for when a field is updated. The daily table does not need to include a row for every issue_date_day after an issue has been Closed for 30+ days, but if it is updated again, then it should record a new date_day row with the newly updated field values.

After the update, it does not need to record the next 30 days unless the issue was re-opened (but whatever you all think is best in this case works for me).

dbt Project configurations

version: '1.3.1'
config-version: 2

  jira:
    schema: dbt
  jira_source:
    schema: dbt_stg

  jira_database: raw
  jira_schema: jira 

  jira:
    issue_field_history_columns: [
      'Affects versions'
      ,'Approved By'
      ,'Assignee'
      ,'Battery Serial Number'
      ,...
      ,'Zip Number'
     ]

Package versions

packages:
  - package: dbt-labs/dbt_utils
    version: 1.1.0

  - package: fivetran/jira
    version: [">=0.13.0", "<0.14.0"]

What database are you using dbt with?

snowflake

dbt Version

Core:
  - installed: 1.3.1
  - latest:    1.5.0 - Update available!

Additional Context

Example of problematic output:

Corresponding issue's history in the Jira UI:

Are you willing to open a PR to help address this issue?

[ ] Yes.
[X] Yes, but I will need assistance and will schedule time during our office hours for guidance
[ ] No.

fivetran-joemarkiewicz commented 1 year ago

Hi @kenzie-marsh thanks so much for opening this issue!

I believe I have an understanding of what may be occurring here. I will plan to do some more investigating, but my team and I will be sure to pick this up at the start of our next sprint (next Wed) and will be sure to share updates as we work through addressing the bug.

Thanks again!

fivetran-joemarkiewicz commented 1 year ago

This bugfix is now live in the v0.14.0 release of the Jira package! As such, I will close out this issue. Please feel free to reopen if the issue persists after upgrading.

Thanks again for all your help @kenzie-marsh!

fivetran / dbt_jira