fivetran / dbt_jira

Data models for Fivetran's Jira connector built using dbt.
https://fivetran.github.io/dbt_jira/
Apache License 2.0
8 stars 13 forks source link

[Feature] <Add optional Project Configuration for specifying Jira Projects> #89

Closed jon-openspace closed 1 year ago

jon-openspace commented 1 year ago

Is there an existing feature request for this?

Describe the Feature

We use Jira for software engineering work and for operations work. I would like to use this package to analyze the software engineering tickets, but not the operations tickets. These tickets are separated by projects.

Describe alternatives you've considered

I could use dbt to create copies of the jira tables that only contain tickets in the software engineering project.

Are you interested in contributing this feature?

Anything else?

No response

fivetran-joemarkiewicz commented 1 year ago

Hi @jon-openspace thanks for opening this feature request!

From your description, it seems like you would prefer to enhance the issue end models to include a project field. This would allow you to filter for only issues relating to the projects you are wishing to do analysis towards. Is this a correct assumption?

jon-openspace commented 1 year ago

Hi @fivetran-joemarkiewicz!

Not exactly, and it looks like the end models already include Project Fields?

I actually want to exclude certain projects from the top-level staging models. We have certain Jira Projects that contain an order of magnitude more data than our Software Engineering project, and I feel like running those issues through the transformations in this package would just extend the time for dbt build.

I will try running this package with and without the exclusions to see how they impact runtime.

fivetran-joemarkiewicz commented 1 year ago

@jon-openspace thanks for this additional context. I was unaware you were looking to do this filtering at the staging models. Unfortunately, I don't think excluding them in the end model will do much to help with the performance as they are still brought all the way through to the end model.

However, you may be able to make this custom edit yourself in the upstream staging model. If you wanted, you could follow our instructions here regarding how to override the logic in an upstream model from our package. My thoughts are, you could override the stg_jira__issue model to include a filter at the final select statement that only includes project_ids for your engineering project. This should filter out any extra issues and reduce the build time as the stg_jira__issue model is the core table that fuels the majority of the end models.

jon-openspace commented 1 year ago

@fivetran-joemarkiewicz Thanks! I didn't realize there was a way to override package default models - I'll bookmark that writeup and share it with my team. Overriding package default models makes a lot more sense than including filters as configuration options. I'll try out that method.