This introduces a way to build the renamed events (from stg_events__all) model incrementally.
Building this model incrementally has a positive impact on all other models in this package. We provide a single configuration option for tuning the look-back window fullstory_incremental_interval. When merging new data into the incremental model, we will look back up to fullstory_incremental_interval in the events table in order to compare data. This configuration option can impact cost greatly (larger look back window = more data processed for each run).
As a result of building the events model incrementally, we must move any aggregation downstream (int_devices > devices, int_identifies > identifies). If we were to build an aggregated model incrementally, there is an opportunity for an error in calculation.
TODO
[x] Add docs for columns in events, devices, identities.
This introduces a way to build the renamed
events(fromstg_events__all) model incrementally.Building this model incrementally has a positive impact on all other models in this package. We provide a single configuration option for tuning the look-back window
fullstory_incremental_interval. When merging new data into the incremental model, we will look back up tofullstory_incremental_intervalin theeventstable in order to compare data. This configuration option can impact cost greatly (larger look back window = more data processed for each run).As a result of building the
eventsmodel incrementally, we must move any aggregation downstream (int_devices>devices,int_identifies>identifies). If we were to build an aggregated model incrementally, there is an opportunity for an error in calculation.TODO
events,devices,identities.target_*fields.