This changes our highly used models to the materialization type incremental so that we only update rows that need to be updated when new data comes in.
In order to make this efficient, I had to change stg_events__all to also be incremental and add some partitioning / clustering logic. This change alone makes the life of all the down stream tables much easier.
Now users, and sessions are incrementally loaded which results in a dramatic decrease of processed bytes for each dbt run. Internally, I have measured this at a > 99% reduction in the amount of processing required to load new data.
This changes our highly used models to the materialization type
incrementalso that we only update rows that need to be updated when new data comes in.In order to make this efficient, I had to change
stg_events__allto also be incremental and add some partitioning / clustering logic. This change alone makes the life of all the down stream tables much easier.Now
users, andsessionsare incrementally loaded which results in a dramatic decrease of processed bytes for each dbt run. Internally, I have measured this at a > 99% reduction in the amount of processing required to load new data.