search

fivetran / dbt_shopify

Fivetran's Shopify dbt package
https://fivetran.github.io/dbt_shopify/
Apache License 2.0
52 stars 40 forks source link

[Feature] Set Models to Incremental Materializations #65

Closed brandonhauer closed 3 months ago

brandonhauer commented 11 months ago

Is there an existing feature request for this?

Describe the Feature

Adapter: Snowflake Would love an easy way to set the output tables to refresh incrementally. Currently, tables are replaced every time a dbt job is run. Because of this, I am unable to create snowflake streams or dynamic tables that reference these tables as a source. If the tables were refreshed incrementally, the links to these dynamic tables would not break.

Describe alternatives you've considered

No response

Are you interested in contributing this feature?

Anything else?

No response

elanfivetran commented 10 months ago

Hey Brandon, thanks for opening up this feature request and your engagement with our Shopify models!

I definitely can understand the benefits of implementing incremental models within our package. However, adding so would be a large engineering effort on our end and we haven't seen the demand to justify the addition of incremental models to our non-historical output models yet. That being said it is a feature we would open to exploring in the future if we see increased demand!

gazjones00 commented 7 months ago

@elanfivetran, this is definitely something we would benefit from too. Our tables consist of huge amounts of data which cause latency when running these DBT jobs. Providing Incremental models would be incredibly useful and help to reduce latency

fivetran-catfritz commented 3 months ago

@brandonhauer and @gazjones00 In the most recent release v0.12.0 (see changelog for full details), we added incremental support for the models that tend to have a higher row-count, which models are:

I am closing this issue, but please feel free to reach out to us in this thread with any feedback or questions!