Not all models include the enabled config

[santina-brohen]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

Currently, the package expects certain sources to be available. Some models, but not all, include the enabled config, which makes it possible to skip the model if the source is not available. For example, stg_zuora__credit_balance_adjustment includes the following: {{ config(enabled=var('zuora__using_credit_balance_adjustment', true)) }} (code here). Of the 19 models, only 4 can be disabled:

• stg_zuora__credit_balance_adjustment
• stg_zuora__refund
• stg_zuora__refund_invoice_payment
• stg_zuora__taxation_item

At Pleo, we are not ingesting all the sources that are expected by the package, so even when leveraging the enabled config in the models where it's available, we are still left with models raising errors.

With the changes in this PR, it is possible to disable any model, not just the subset listed above. Having the ability to disable models for which the sources are not ingested via Fivetran is a great feature, and it opens up use of this package even when only a few Fivetran sources are available.

Relevant error log or model output

No response

Expected behavior

I would expect to be able to leverage the package for as many models as we currently ingest via Fivetran. IOW, I would expect to be able to configure each model to be enabled.

Before making the changes, we had four models that raised errors because the sources are not available. I ran dbt build --select +zuora_source and the following shows the error:

After making the changes and disabling the models for sources we do not ingest, I ran dbt build --select +zuora_source again; no errors were raised.

I then removed one of the new variables to ensure that it still correctly raised an error; it did:

Note: zuora__using_contact: false is commented out.

dbt Project configurations

  zuora__using_credit_balance_adjustment: false
  zuora__using_refund: false
  zuora__using_refund_invoice_payment: false
  zuora__using_taxation_item: false

Package versions

  - package: fivetran/zuora_source
    version: [">=0.2.0", "<0.3.0"]

What database are you using dbt with?

bigquery

dbt Version

Additional Context

I made and tested the changes in this PR.

Are you willing to open a PR to help address this issue?

• [X] Yes.
• [ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-catfritz]

Hi @santina-brohen, thanks for reaching out! To add some context on the decision-making process regarding the dbt_zuora_source models that can be disabled, we chose source tables that aren't frequently synced. We also chose tables that even if disabled, they still contribute to meaningful models in the downstream transform package dbt_zuora. We've also limited the tables available for disablement due to the maintenance implications downstream and also since certain tables are crucial for the functionality of our downstream models.

Now, regarding the issue you've encountered, thank you for sharing details about your setup. It appears that you haven't synced several tables. While we prefer not to add enablement configurations for every model, understanding more about your use case would be beneficial. Could you elaborate on your primary uses for our packages? Additionally, we're interested in why you consider the tables you've disabled as unnecessary. This insight will help understand what other options there might be.

[santina-brohen]

Hi @santina-brohen, thanks for reaching out! To add some context on the decision-making process regarding the dbt_zuora_source models that can be disabled, we chose source tables that aren't frequently synced. We also chose tables that even if disabled, they still contribute to meaningful models in the downstream transform package dbt_zuora. We've also limited the tables available for disablement due to the maintenance implications downstream and also since certain tables are crucial for the functionality of our downstream models.

Hello, @fivetran-catfritz! 👋🏼 Thanks for the context and explaining! That makes sense. (FWIW, if my PR was accepted, I planned to mirror the changes in the transformation package, understanding of the dependency.)

Now, regarding the issue you've encountered, thank you for sharing details about your setup. It appears that you haven't synced several tables. While we prefer not to add enablement configurations for every model, understanding more about your use case would be beneficial. Could you elaborate on your primary uses for our packages? Additionally, we're interested in why you consider the tables you've disabled as unnecessary. This insight will help understand what other options there might be.

At Pleo, we are migrating to Zuora to manage our subscriptions. As part of the process, we need to migrate our dbt models to the new sources for key metrics like MRR. We are ingesting only those sources needed, which at this point includes the following:

• account
• credit_memo
• invoice
• invoice_adjustment
• invoice_item
• invoice_item_adjustment
• product_rate_plan
• product_rate_plan_charge
• product_rate_plan_charge_tier
• rate_plan
• rate_plan_charge
• rate_plan_charge_tier
• product
• subscription
• usage

Our primary use case would be to leverage the package to generate staging models on which we will build SSOTs in dbt. Without being able to disable certain models from the transformation package, we wouldn't be able to use it, however, I evaluated it anyway. It does not work for our needs due to a difference in definitions/calculation of MRR from what is applied in zuora__monthly_recurring_revenue.

In addition to the package expecting sources that we do not ingest, as you may have noticed from the list above, we are ingesting sources that are not currently covered by the package. In thinking through the best path forward, as a first step, we decided to try to contribute to the package to make it more flexible and cover other use cases for which not all sources are being ingested. If there was an appetite for these types of contributions, then the next step would be to add models for the sources currently not covered in the package as it seems reasonable that others would like them too.

If we can't leverage the package, then we will simply source the models in our dbt staging layer as we do with others. We decided against maintaining our own fork of the package.

Does that help answer your question? If not, please let me know!

[fivetran-catfritz]

Hi @santina-brohen Thank you for the additional information! Your use case is interesting, and it's likely that there are other users with similar subscription-only usage patterns. I am relaying this to our product team. If they express interest in implementing such an update, I'll loop you into the next step of the process.

In the meantime, could you please expand on the variances in your MRR calculations? We're continually seeking ways to enhance our models, and your insights could be valuable in refining our approach!

[fivetran-catfritz]

Hi @santina-brohen following up to my last message, we are still interested in the use case of subscriptions only, however since this would require a fairly large update to this package, we will not be able to complete this in the immediate quarter. In the meantime, we will continue to keep you updated on the status and progress of any updates in this thread!