[Feature] Understanding Snowflake and tmp view models

fivetran / dbt_hubspot_source

Data models for Hubspot built using dbt.

https://fivetran.github.io/dbt_hubspot_source/

Apache License 2.0

31 stars 30 forks source link

[Feature] Understanding Snowflake and tmp view models #116

Closed fivetran-catfritz closed 10 months ago

fivetran-catfritz commented 11 months ago

Is there an existing feature request for this?

[X] I have searched the existing issues

Describe the Feature

Continuing discussions from this slack thread and this one, this issue is for a deeper investigation into this.

The problem arises in Snowflake when a source's dimensions changes, but the dbt model has been run again.

The proposed solution is to update our tmp view models is to use dbt_utils.star at least for Snowflake targets.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

[X] Yes.
[ ] Yes, but I will need assistance and will schedule time during your office hours for guidance.
[ ] No.

Anything else?

No response

reubster commented 11 months ago

"The problem arises in Snowflake when a source's dimensions changes, but the dbt model has not yet been run again."

@fivetran-catfritz .. responding to your Slack question;

"In Snowflake, are you also persisting docs, or does this happen regardless?"

Went to look up what you meant by "persisting docs". I didn't know that was a feature in dbt. No, we're not using it.

When I look at the view definition in Snowflake, Snowflake has 'materialised' the column names of the view... but still has the same select * from body.

Seems a straightforward fix to itemise the columns at dbt-run using dbt_utils.star

fivetran-joemarkiewicz commented 11 months ago

Hi @reubster I would agree with you that we can ideally solve this with a dbt_utils.star in the tmp models.

My team is planning to pick this up for hubspot in the coming sprint (starting 10/18). However, if you need an immediate fix for this I noticed you are open to creating a PR. If you would like to open a PR I would be more than happy to review your PR and integrate your changes into the next release of the HubSpot package!