[Bug] Source Freshness tests still execute for sources that are distabled

[epapineau]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the issue

We installed the fivetran_log package and disabled additional tables with the recommending configuration in dbt_project.yml.

When our next prod job executed dbt source freshness we received object does not exist errors for the tables that were disabled.

Relevant error log or model output

23:45:38  Database Error in source account_membership (models/staging/src_fivetran_log.yml)
23:45:38    002003 (42S02): SQL compilation error:
23:45:38    Object 'RAW.FIVETRAN_LOG.ACCOUNT_MEMBERSHIP' does not exist or not authorized.
23:45:38  
23:45:38  Database Error in source destination_membership (models/staging/src_fivetran_log.yml)
23:45:38    002003 (42S02): SQL compilation error:
23:45:38    Object 'RAW.FIVETRAN_LOG.DESTINATION_MEMBERSHIP' does not exist or not authorized.
23:45:38  
23:45:38  Database Error in source user (models/staging/src_fivetran_log.yml)
23:45:38    002003 (42S02): SQL compilation error:
23:45:38    Object 'RAW.FIVETRAN_LOG.USER' does not exist or not authorized.
23:45:38  Done.

Expected behavior

If we've communicated we do not have the tables in our dataset, we should not try to run source freshness for them0.6.1

dbt Project configurations

vars:
  fivetran_log:
    fivetran_log_database: raw
    fivetran_log_schema: fivetran_log
    fivetran_log_using_transformations: false  
    fivetran_log_using_triggers: false  
    fivetran_log_using_account_membership: false 
    fivetran_log_using_destination_membership: false 
    fivetran_log_using_user: false  

Package versions

0.6.1

What database are you using dbt with?

snowflake

dbt Version

1.1

Additional Context

No response

Are you willing to open a PR to help address this issue?

• [ ] Yes.
• [ ] Yes, but I will need assistance and will schedule time during our office hours for guidance
• [ ] No.

[fivetran-joemarkiewicz]

Hi @epapineau! 👋

Thanks for reaching out and creating this issue. The logic of why this issue occurs for you does make sense to me. The package disables components only at the model level, and not at the source. As such, I can see how the dbt run works as intended (because the package has properly disabled those unwanted models) but then the dbt source freshness fails because the sources do not exist and there is nothing done within the package to account for that. 😿

However, we actually have implemented a workaround in a few of our other packages that leverages a custom freshness test (which overrides the default test within dbt core) to allow for the freshness tests of these absent sources to be ignored. See how we apply the logic in our dbt_zendesk_source package. Therefore, we can apply the same logic to these sources within this package.

Another avenue would be to leverage the overrides feature within dbt that would allow you to customize the source and disable the freshness tests altogether.

[epapineau]

@fivetran-joemarkiewicz Thanks for your response! We're hoping to use source freshness here to support dashboard status tiles to communicate fivetran's logging to our Looker users. That means disabling at the source level won't work in our case.

Would you welcome a PR applying the same logic as the dbt_zendesk_source package you linked to this package?

[fivetran-joemarkiewicz]

That sounds great @epapineau and we would absolutely welcome a PR with this same logic!

Let me know if you need any assistance when creating the PR 😄

[epapineau]

@fivetran-joemarkiewicz I gave it a shot using the zendesk repo as an example and am no longer receiving the source freshness errors 🥳

[fivetran-joemarkiewicz]

@epapineau that is great to hear and thank you so much for taking the time to open the PR! 🎉

I will review the PR later today and let you know if I have any suggestions before moving forward.

[fivetran-joemarkiewicz]

Thanks again so much for your contribution @epapineau 🥇

These changes are now available in the v0.6.2 release of the dbt_fivetran_log package. You can expect this to be live on the dbt hub at the top of the hour. Please feel free to open a new issue if you encounter anything else. 😄