Closed wuservices closed 1 year ago
Hi @wuservices thanks for opening this Issue as well as Issue #18!
This is actually a request we have seen come through on our dbt_hubspot package and have been able to institute a relatively low impact fix for these disabled tables. You can see the fix we did here. I will be able to easily add this feature to the package. However, I will focus first on upgrading our dbt_zendesk package for dbt 20 compatibility first so there are no compatibility issues between this source package and the transform package.
hey @wuservices our own data is in line with the pre-existing freshness tests, so i am thinking that the ideal solution here may be for you to override the zendesk source and set the organization_tag
freshness to null
however, worth noting that dbt 0.21.0 will introduce --exclude
for freshness tests :eyes: (see PR and pre-release notes)
hi @fivetran-jamie thanks for the tips! @fivetran-joemarkiewicz pointed out the same freshness override in office hours, and that was a huge help! We were able to get our freshness tuned up nicely with that, so we're OK.
However, we did also discuss that good defaults are always nice when possible. For the benefit of others, I'd be curious why you'd have freshness by default on organization_tag
, but no freshness for organization
? For many companies using Zendesk, I could imagine that the clients they work with organization
may not change often (or never at all for B2C). This seems to be reflected in your default disabled freshness for organization
. However, if organization_tag
has freshness, does that mean you'd assume that people would be changing tags on a regular basis? Although we don't use those, I'd imagine that most people would have stable tags.
that's a good point that joe and i were honestly unsure about. in our own zendesk data, organization
doesn't change too often, but it does appear that we have fresh tags on them. i'll look into this more and talk to our internal customer success folks who are using zendesk (and maybe do a quick poll in dbt slack or something) to see if this manual tagging is actually happening often.
definitely aiming for the nicest defaults possible and really appreciate your input! i think it'd be good to keep this issue open and mark it as won't fix
for now in case other customers are coming across the same staleness issues with this table. glad to hear the override works for you for the time being 🙂
Closing this issue as it is a wontfix.
Are you a current Fivetran customer? Jon Wu, Architect, Paper Culture
Describe the bug
zendesk.organization_tag
fails freshness checks. It appears that it shouldn't be checked.Steps to reproduce
dbt source snapshot-freshness
ERROR STALE freshness of zendesk.organization_tag
Expected behavior
No freshness error should occur if data sync is working. In my case, it appears that this data just doesn't get updated very often. I noticed that
freshness: null
is configured forzendesk.organization
. Should that just be copied toorganization_tag
to ignore freshness here too?Project variables configuration
Package Version
Warehouse
- [x] BigQuery - [ ] Redshift - [ ] Snowflake - [ ] Postgres - [ ] Databricks - [ ] Other (provide details below) **Additional context** Freshness is ignored for `organization`: https://github.com/fivetran/dbt_zendesk_source/blob/master/models/src_zendesk.yml#L130 **Screenshots**Please indicate the level of urgency
Testing for the first time, but hopefully this is a simple fix.
Are you interested in contributing to this package?