search

fivetran / dbt_quickbooks_source

Data models for QuickBooks built using dbt.
https://fivetran.github.io/dbt_quickbooks_source/
Apache License 2.0
5 stars 18 forks source link

QUESTION - should "using_X" configuration be overridden by models: enabled: configuration in dbt_project.yml? #20

Closed raubrey2014 closed 1 year ago

raubrey2014 commented 2 years ago

Are you a Fivetran customer? Yes! - Ryan Aubrey, software engineer, Outlast

Your Question Hi there! My company is using this package to manage our Quickbooks transformations in our data warehouse.

In Quickbooks, we do not use the Refund Receipt functionality - setting the environment variable using_receipt_refund works great to pass over the models dependent on refund receipts! Until we need to enable/disable the quickbooks models overall per dev/prod environment that is...

TLDR: using_X var is being overridden. Do you have any suggestions on how we can get the intended functionality?

Context

vars:
  using_refund_receipt: false

models:
  quickbooks:
    enabled: "{{ (target.name == 'prod') | as_bool }}"

  quickbooks_source:
    enabled: "{{ (target.name == 'prod') | as_bool }}"

As you can see, we want to disable the refund receipt module always, as well as only ever enable quickbooks models in our "prod" environment. The above configuration gives precedence to the models>quickbooks>enabled: true when in our production environment.

This means that with the above configuration, when deploying our models to our production environment dbt still tries to deploy the models related to refund receipts.

Do you have any suggestions on how we can get the intended functionality?

Solutions I tried

I tried disabling individual models, but the list is.. long :) 

models:
  quickbooks:
    enabled: "{{ (target.name == 'prod') | as_bool }}"

  quickbooks_source:
    enabled: "{{ (target.name == 'prod') | as_bool }}"
     stg_quickbooks__refund_receipt_tmp:
       enabled: false
     stg_quickbooks__refund_receipt:
       enabled: false
    ...

Any help here is very much appreciated!

fivetran-joemarkiewicz commented 2 years ago

Hey @raubrey2014 👋 I am glad to see you are using our QuickBooks package! I have actually encountered a similar question to this, and found a workaround (however we will want to determine a more scalable and sustainable solution in the future). What I would recommend for the time being is the following:

You won't want to enable/disable the models themselves, since the source and transform package relies on conditionals scattered throughout to use or not use the relevant source tables based on the variable declarations. Therefore, we will want to dynamically change the variables instead.

However, I have been trying to get your dynamic variable switch to work, but continue receiving the below error:

at path ['enabled']: 'False' is not of type 'boolean'

Let me look into this some more and try to find a solution for you. I imagine this is a behavior other users of this package will want to know about. Hopefully we can get this working!

fivetran-joemarkiewicz commented 2 years ago

I @raubrey2014 I just wanted to post back that I have been unable to find a solid solution for this at the moment. Have you by chance found any other workaround since posting this question?

fivetran-joemarkiewicz commented 1 year ago

Closing this issue as it has gone a bit stale since it's creation. Please feel free to re-open if you feel there is more to discuss.