[Bug] Status field being populated with field options

[austensd]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

After upgrading from version >=0.8.0, and running a full refresh of the Jira data, the status field on both jira__daily_issue_field_history and jira__issue_enhanced is being populated with values from the jira.field_options table, instead of the status id it was previously, or a value from the jira.status table.

Relevant error log or model output

No response

Expected behavior

I would expect the status field to either contain the status_id, or the name from the jira.status table.

dbt Project configurations

version: '1.0.0'
config-version: 2
vars:
  jira:
    issue_field_history_columns: ['Work Type', 'Story Points', 'Squad']

Package versions

packages:

• package: fivetran/jira version: [">=0.8.0", "<0.9.0"]

What database are you using dbt with?

redshift

dbt Version

1.0.7

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @austensd thanks so much for opening this issue. That definitely seems strange and something we should take a closer look at. Would you mind sharing an example or two of this behavior?

[austensd]

Thanks for the quick response, here are some samples from our warehouse. The first image shows a sample of the issue_id, status_id, and status fields for some records from our jira__issue_enhanced the second shows some of the records from our jira.status table and the third shows a sample from our jira.field_option table.

If we look at the status_id of 10035, the name from the status table should be To Do, but is instead showing as Internet Explorer, which is the field option that has the overlapping id.

Hopefully this helps.

[fivetran-joemarkiewicz]

Thanks so much for the quick turnaround @austensd! I have a quick follow up question - is this only occurring for the status field?

If so, I think the issue may be occurring in the snippet below https://github.com/fivetran/dbt_jira/blob/b383b19120efadff88abf00ed522fd5527976d93/models/jira__daily_issue_field_history.sql#L98-L102

I wonder if we simply add status to the not in category, it may resolve the error. If you make that change on your end do you see the status' update properly?

[fivetran-joemarkiewicz]

Now that I look at the above, there seem to be other parts of the model where we may need to adjust the logic to exclude status from this operation. I will continue to look into this on my end 👀

[austensd]

status is the only field I've noticed an issue with, in terms of the other fields on the jira__daily_issue_field_history, we are not currently using the sprint field, and all of our custom fields appear to be correct. And I cannot see any issues with any other field on the jira__issue_enhanced table.

[fivetran-joemarkiewicz]

Thanks for the update @austensd! I will be OOO the remainder of the week, but will come back to this ticket when I return on Tuesday.

[fivetran-joemarkiewicz]

@austensd I just wanted to post back that I am still investigating this issue. I will be sure to post back once I identify a possible solution.

[fivetran-joemarkiewicz]

Hi @austensd I apologize for the delayed response, but we think we may have identified a solution. We will start working on the fix in coming week and will share a working version of the package for you to try. If all goes well there, we can include the changes in our next release!

[jamienewcomb]

Hey @fivetran-joemarkiewicz

Noticed I am finding this issue aswell after creating some dashboards in tableau. using the same table and finding the status field being populated with random values. Should just be the status id

How are you coming along in implementing a solution? :)

[fivetran-joemarkiewicz]

Hi @jamienewcomb thanks for reaching out and sharing that you are experiencing this as well. I should have a working branch early next week that I can share for you to test out and confirm if the issue still persists.

I hope to get this fix addressed by the middle of next week.

[jamienewcomb]

Thanks for the update @fivetran-joemarkiewicz OOI: Do you know of anyone (of where I could ask) who has managed to create a burndown chart using this table (jira__daily_issue_field_history)? Have managed everything else but struggling with this one. Thanks!

[fivetran-joemarkiewicz]

Hi All,

I have been able to get a working version of the package that should address this issue. The changes are linked in the PR above. It would be great if you would be able to test the version of the package with these changes and let me know if the issue still persists. To test these changes you can use the following config in your packages.yml

packages:
  - git: https://github.com/fivetran/dbt_jira.git
    revision: bugfix/status-correction
    warn-unpinned: false

Please note this will be a breaking change since adjustments have been made to the incremental model. As such, you will need to run a dbt run --full-refresh in order for the changes to be reflected in your warehouse. Let me know how it goes!

Additionally, @jamienewcomb

Do you know of anyone (of where I could ask) who has managed to create a burndown chart using this table (jira__daily_issue_field_history)? Have managed everything else but struggling with this one. Thanks!

Unfortunately, I do not know of any current implementations of this. There may be some folks in dbt Slack that have experience putting this together!

[jamienewcomb]

Thanks @fivetran-joemarkiewicz . I will pass this onto our data team and let you know if it fixes the issue

[fivetran-joemarkiewicz]

@jamienewcomb and @austensd I had found an issue with the previous state of the branch I shared. However, I know feel I have been able to address it and the current build of the branch I shared above should be working as intended.

Let me know if this addresses your issues.

[jamienewcomb]

Thank you @fivetran-joemarkiewicz

It's on my data teams backlog so I am hopeful they can test this within the next week

[fivetran-joemarkiewicz]

This issue has since been addressed in the latest release of the package. Feel free to upgrade the package to see the updates. If the issue still persists please reopen this issue.