search

fivetran / dbt_iterable_source

Fivetran data source models for Iterable using dbt.
https://fivetran.github.io/dbt_iterable_source/
Apache License 2.0
2 stars 5 forks source link

[Bug] Source Freshness executing on unused tables #36

Open justin-fundrise opened 5 days ago

justin-fundrise commented 5 days ago

Is there an existing issue for this?

Describe the issue

https://github.com/fivetran/dbt_iterable_source/blob/847b282848a98f7ea8947df4f0aa5639649845dc/models/src_iterable.yml#L11-L13

While implementing the iterable package, I am running into an error on my slim ci dbt job.

during the execution of dbt source freshness I am encountering the error

19:57:17 13 of 13 ERROR freshness of iterable.user_unsubscribed_channel_history ......... [ERROR in 0.23s]
19:57:17  
19:57:17
19:57:17  Finished running 13 sources in 0 hours 0 minutes and 3.66 seconds (3.66s).
19:57:18  
19:57:18
19:57:18    Database Error in source user_unsubscribed_channel_history (models/src_iterable.yml)
  002003 (42S02): SQL compilation error:
  Object 'db_name.SRC_ITERABLE.USER_UNSUBSCRIBED_CHANNEL_HISTORY' does not exist or not authorized.

Which is correct because I do not have the table USER_UNSUBSCRIBED_CHANNEL_HISTORY

I would expect source freshness to skip this table if it is not explicitly being used.

Relevant error log or model output

19:57:18    Database Error in source user_unsubscribed_channel_history (models/src_iterable.yml)
  002003 (42S02): SQL compilation error:
  Object 'BLUEPRINTS_PROD.SRC_ITERABLE.USER_UNSUBSCRIBED_CHANNEL_HISTORY' does not exist or not authorized.


### Expected behavior

I would expect source freshness to skip tables that are not being used. 

### Possible solution

_No response_

### dbt Project configurations

iterable: +tags: ft_iterable +schema: fivetran # this schema name is prefixed to iterable intermediate: +schema: fivetran # this schema name is prefixed to int_iterable +tags: ft_iterable iterable_source: +schema: fivetran # this schema name is prefixed to stg_iterable +tags: ft_iterable_src

vars: iterable_database: blueprints_prod iterable_schema: src_iterable iterable__using_campaign_label_history: false iterable__using_user_unsubscribed_message_type: false


### Package versions

What database are you using dbt with?

snowflake

How are you running this dbt package?

dbt Cloud™

dbt Version

versionless

Additional Context

No response

Are you willing to open a PR to help address this issue?

fivetran-avinash commented 5 days ago

Hi @justin-fundrise ,

try setting this config in a new src_iterable.yml file. That should exclude a source from freshness calculations.

sources:
  - name: iterable
     overrides: iterable_source
     description: override Iterable package configs

...

  config
     tables: 
       - name: user_unsubscribed_channel_history
         freshness: null
fivetran-avinash commented 3 days ago

Glad to see that you confirmed this worked out @justin-fundrise !

Just wanted to check in--do you think the source freshness tests are important for you? Our team is considering removing them because maintaining these tests is a bit of a pain to maintain and it can lead to errors that you experience, and require these custom configurations.

Let us know your thoughts!

justin-fundrise commented 3 days ago

I think it would be okay to remove the freshness checks. This is something that is more specific to a business’s needs depending on their use case of the data.