search

fivetran / dbt_jira

Data models for Fivetran's Jira connector built using dbt.
https://fivetran.github.io/dbt_jira/
Apache License 2.0
8 stars 15 forks source link

Allow exclusion of priority data #55

Closed SteadyGiant closed 2 years ago

SteadyGiant commented 2 years ago

What change(s) does this PR introduce?

My team doesn't use Priority in Jira, the API returns nothing from our Priority endpoint, and so source jira.priority never gets loaded into the warehouse. Instead, we use custom fields. We still need downstream models like jira__issue_enhanced, which depend on priority data. This PR allows exclusion of priority data, so jira__issue_enhanced can get built even if a team lacks priority data.

Makes priority data optional. Allows new env var jira_using_priorities. Models jira__issue_enhanced and int_jira__issue_join won't require source jira.priority or contain priority-related columns if jira_using_priorities: false.

Did you update the CHANGELOG?

Does this PR introduce a breaking change?

Priority data is still used by default (env var jira_using_priorities: true in dbt_project.yml).

Did you update the dbt_project.yml files with the version upgrade (please leverage standard semantic versioning)? (In both your main project and integration_tests)

Project versions bumped in both files.

Is this PR in response to a previously created Bug or Feature Request

How did you test the PR changes?

Tested manually with dbt CLI (dbt run --select jira_source jira) from my local machine (Windows 10, PowerShell) to a Redshift warehouse using dbt-core 1.0.4 and dbt-redshift 1.0.0. All models defined in this repo are built successfully. All models in fivetran/jira_source package except stg_jira__priority_tmp and stg_jira__priority are built successfully; stg_jira__priority_tmp throws an error and stg_jira__priority gets skipped, but this is intentional.

Output:

> dbt run --select jira_source jira
17:38:22  Running with dbt=1.0.4
17:38:25  Found 130 models, 117 tests, 0 snapshots, 0 analyses, 586 macros, 0 operations, 0 seed files, 37 sources, 0 exposures, 0 metrics
17:38:25
17:38:29  Concurrency: 1 threads (target='dev')
17:38:29
17:38:29  1 of 46 START view model stg_jira.stg_jira__comment_tmp......................... [RUN]
17:38:29  1 of 46 OK created view model stg_jira.stg_jira__comment_tmp.................... [CREATE VIEW in 0.60s]
17:38:29  2 of 46 START view model stg_jira.stg_jira__component_tmp....................... [RUN]
17:38:30  2 of 46 OK created view model stg_jira.stg_jira__component_tmp.................. [CREATE VIEW in 0.59s]
17:38:30  3 of 46 START view model stg_jira.stg_jira__field_option_tmp.................... [RUN]
17:38:30  3 of 46 OK created view model stg_jira.stg_jira__field_option_tmp............... [CREATE VIEW in 0.56s]
17:38:30  4 of 46 START view model stg_jira.stg_jira__field_tmp........................... [RUN]
17:38:31  4 of 46 OK created view model stg_jira.stg_jira__field_tmp...................... [CREATE VIEW in 0.60s]
17:38:31  5 of 46 START view model stg_jira.stg_jira__issue_field_history_tmp............. [RUN]
17:38:32  5 of 46 OK created view model stg_jira.stg_jira__issue_field_history_tmp........ [CREATE VIEW in 0.59s]
17:38:32  6 of 46 START view model stg_jira.stg_jira__issue_link_tmp...................... [RUN]
17:38:32  6 of 46 OK created view model stg_jira.stg_jira__issue_link_tmp................. [CREATE VIEW in 0.60s]
17:38:32  7 of 46 START view model stg_jira.stg_jira__issue_multiselect_history_tmp....... [RUN]
17:38:33  7 of 46 OK created view model stg_jira.stg_jira__issue_multiselect_history_tmp.. [CREATE VIEW in 0.60s]
17:38:33  8 of 46 START view model stg_jira.stg_jira__issue_tmp........................... [RUN]
17:38:33  8 of 46 OK created view model stg_jira.stg_jira__issue_tmp...................... [CREATE VIEW in 0.60s]
17:38:33  9 of 46 START view model stg_jira.stg_jira__issue_type_tmp...................... [RUN]
17:38:34  9 of 46 OK created view model stg_jira.stg_jira__issue_type_tmp................. [CREATE VIEW in 0.58s]
17:38:34  10 of 46 START view model stg_jira.stg_jira__priority_tmp....................... [RUN]
17:38:34  10 of 46 ERROR creating view model stg_jira.stg_jira__priority_tmp.............. [ERROR in 0.32s]
17:38:34  11 of 46 START view model stg_jira.stg_jira__project_tmp........................ [RUN]
17:38:35  11 of 46 OK created view model stg_jira.stg_jira__project_tmp................... [CREATE VIEW in 0.57s]
17:38:35  12 of 46 START view model stg_jira.stg_jira__resolution_tmp..................... [RUN]
17:38:36  12 of 46 OK created view model stg_jira.stg_jira__resolution_tmp................ [CREATE VIEW in 0.58s]
17:38:36  13 of 46 START view model stg_jira.stg_jira__sprint_tmp......................... [RUN]
17:38:36  13 of 46 OK created view model stg_jira.stg_jira__sprint_tmp.................... [CREATE VIEW in 0.57s]
17:38:36  14 of 46 START view model stg_jira.stg_jira__status_category_tmp................ [RUN]
17:38:37  14 of 46 OK created view model stg_jira.stg_jira__status_category_tmp........... [CREATE VIEW in 0.54s]
17:38:37  15 of 46 START view model stg_jira.stg_jira__status_tmp......................... [RUN]
17:38:37  15 of 46 OK created view model stg_jira.stg_jira__status_tmp.................... [CREATE VIEW in 0.57s]
17:38:37  16 of 46 START view model stg_jira.stg_jira__user_tmp........................... [RUN]
17:38:38  16 of 46 OK created view model stg_jira.stg_jira__user_tmp...................... [CREATE VIEW in 0.56s]
17:38:38  17 of 46 START view model stg_jira.stg_jira__version_tmp........................ [RUN]
17:38:38  17 of 46 OK created view model stg_jira.stg_jira__version_tmp................... [CREATE VIEW in 0.57s]
17:38:38  18 of 46 START table model stg_jira.stg_jira__comment........................... [RUN]
17:38:41  18 of 46 OK created table model stg_jira.stg_jira__comment...................... [SELECT in 2.11s]
17:38:41  19 of 46 START table model stg_jira.stg_jira__component......................... [RUN]
17:38:41  19 of 46 OK created table model stg_jira.stg_jira__component.................... [SELECT in 0.84s]
17:38:41  20 of 46 START table model stg_jira.stg_jira__field_option...................... [RUN]
17:38:42  20 of 46 OK created table model stg_jira.stg_jira__field_option................. [SELECT in 0.85s]
17:38:42  21 of 46 START table model stg_jira.stg_jira__field............................. [RUN]
17:38:43  21 of 46 OK created table model stg_jira.stg_jira__field........................ [SELECT in 0.84s]
17:38:43  22 of 46 START table model stg_jira.stg_jira__issue_field_history............... [RUN]
17:38:52  22 of 46 OK created table model stg_jira.stg_jira__issue_field_history.......... [SELECT in 9.19s]
17:38:52  23 of 46 START table model stg_jira.stg_jira__issue_link........................ [RUN]
17:38:53  23 of 46 OK created table model stg_jira.stg_jira__issue_link................... [SELECT in 0.84s]
17:38:53  24 of 46 START table model stg_jira.stg_jira__issue_multiselect_history......... [RUN]
17:38:56  24 of 46 OK created table model stg_jira.stg_jira__issue_multiselect_history.... [SELECT in 3.05s]
17:38:56  25 of 46 START table model stg_jira.stg_jira__issue............................. [RUN]
17:38:58  25 of 46 OK created table model stg_jira.stg_jira__issue........................ [SELECT in 1.88s]
17:38:58  26 of 46 START table model stg_jira.stg_jira__issue_type........................ [RUN]
17:38:59  26 of 46 OK created table model stg_jira.stg_jira__issue_type................... [SELECT in 0.89s]
17:38:59  27 of 46 SKIP relation stg_jira.stg_jira__priority.............................. [SKIP]
17:38:59  28 of 46 START table model stg_jira.stg_jira__project........................... [RUN]
17:39:00  28 of 46 OK created table model stg_jira.stg_jira__project...................... [SELECT in 0.96s]
17:39:00  29 of 46 START table model stg_jira.stg_jira__resolution........................ [RUN]
17:39:01  29 of 46 OK created table model stg_jira.stg_jira__resolution................... [SELECT in 0.88s]
17:39:01  30 of 46 START table model stg_jira.stg_jira__sprint............................ [RUN]
17:39:02  30 of 46 OK created table model stg_jira.stg_jira__sprint....................... [SELECT in 0.98s]
17:39:02  31 of 46 START table model stg_jira.stg_jira__status_category................... [RUN]
17:39:03  31 of 46 OK created table model stg_jira.stg_jira__status_category.............. [SELECT in 0.92s]
17:39:03  32 of 46 START table model stg_jira.stg_jira__status............................ [RUN]
17:39:04  32 of 46 OK created table model stg_jira.stg_jira__status....................... [SELECT in 0.94s]
17:39:04  33 of 46 START table model stg_jira.stg_jira__user.............................. [RUN]
17:39:05  33 of 46 OK created table model stg_jira.stg_jira__user......................... [SELECT in 0.90s]
17:39:05  34 of 46 START table model stg_jira.stg_jira__version........................... [RUN]
17:39:06  34 of 46 OK created table model stg_jira.stg_jira__version...................... [SELECT in 0.93s]
17:39:06  35 of 46 START incremental model test.int_jira__issue_calendar_spine............ [RUN]
17:40:50  35 of 46 OK created incremental model test.int_jira__issue_calendar_spine....... [SELECT in 104.21s]
17:40:50  36 of 46 START incremental model test.int_jira__agg_multiselect_history......... [RUN]
17:40:59  36 of 46 OK created incremental model test.int_jira__agg_multiselect_history.... [SELECT in 8.69s]
17:40:59  37 of 46 START table model test.int_jira__issue_type_parents.................... [RUN]
17:41:02  37 of 46 OK created table model test.int_jira__issue_type_parents............... [SELECT in 3.11s]
17:41:02  38 of 46 START incremental model test.int_jira__combine_field_histories......... [RUN]
17:41:56  38 of 46 OK created incremental model test.int_jira__combine_field_histories.... [SELECT in 53.70s]
17:41:56  39 of 46 START incremental model test.int_jira__daily_field_history............. [RUN]
17:42:01  39 of 46 OK created incremental model test.int_jira__daily_field_history........ [SELECT in 4.83s]
17:42:01  40 of 46 START table model test.int_jira__issue_join............................ [RUN]
17:42:07  40 of 46 OK created table model test.int_jira__issue_join....................... [SELECT in 6.12s]
17:42:07  41 of 46 START incremental model test.int_jira__pivot_daily_field_history....... [RUN]
17:42:09  41 of 46 OK created incremental model test.int_jira__pivot_daily_field_history.. [SELECT in 1.23s]
17:42:09  42 of 46 START table model test.int_jira__field_history_scd..................... [RUN]
17:42:10  42 of 46 OK created table model test.int_jira__field_history_scd................ [SELECT in 1.61s]
17:42:10  43 of 46 START incremental model test.jira__daily_issue_field_history........... [RUN]
17:44:59  43 of 46 OK created incremental model test.jira__daily_issue_field_history...... [SELECT in 168.66s]
17:44:59  44 of 46 START table model test.jira__issue_enhanced............................ [RUN]
17:45:25  44 of 46 OK created table model test.jira__issue_enhanced....................... [SELECT in 25.88s]
17:45:25  45 of 46 START table model test.jira__project_enhanced.......................... [RUN]
17:45:27  45 of 46 OK created table model test.jira__project_enhanced..................... [SELECT in 1.93s]
17:45:27  46 of 46 START table model test.jira__user_enhanced............................. [RUN]
17:45:28  46 of 46 OK created table model test.jira__user_enhanced........................ [SELECT in 1.22s]
17:45:28
17:45:28  Finished running 17 view models, 23 table models, 6 incremental models in 423.05s.
17:45:28
17:45:28  Completed with 1 error and 0 warnings:
17:45:28
17:45:28  Database Error in model stg_jira__priority_tmp (models\tmp\stg_jira__priority_tmp.sql)
17:45:28    relation "jira.priority" does not exist
17:45:28    compiled SQL at target\run\jira_source\models\tmp\stg_jira__priority_tmp.sql
17:45:28
17:45:28  Done. PASS=44 WARN=0 ERROR=1 SKIP=1 TOTAL=46

Select which warehouse(s) were used to test the PR

fivetran-joemarkiewicz commented 2 years ago

@everettttt thanks for opening this PR! Taking a look at the changes you have proposed, I feel this would be a valuable addition to the package.

There is one adjustment that will need to be made in order for the package to properly succeed with this variable. We will need to add the jira_using_priorities config logic to our source models:

Once the config logic is added to these models to be disabled with the jira_using_priorities: false then we will be good to go! Let me know if you would be open to creating a PR on our source package. If not, I am happy to tackle that update 😄

SteadyGiant commented 2 years ago

Let me know if you would be open to creating a PR on our source package. If not, I am happy to tackle that update

I'll do it.

fivetran-joemarkiewicz commented 2 years ago

Thanks so much @everettttt! Feel free to tag me in the PR and I will be sure to review it as soon as possible once it is open 😄

SteadyGiant commented 2 years ago

Just realized I'll have to return to this in 2 weeks, though. If this needs doing before then, feel free, but otherwise I'm willing to do it.

fivetran-joemarkiewicz commented 2 years ago

Thanks for the update @everettttt! 2 weeks will be perfect. There haven't been any other requests at the moment for this to be addressed asap. So that timeline is great. Thanks again 😄

SteadyGiant commented 2 years ago

There is one adjustment that will need to be made in order for the package to properly succeed with this variable. We will need to add the jira_using_priorities config logic to our source models:

Submitted a PR to fivetran/dbt_jira_source.

I'll resolve the conflicts in the present PR tomorrow.

SteadyGiant commented 2 years ago

Conflicts resolved, I hope. Added myself as a contributor to the changelog.