search

snowplow / dbt-snowplow-unified

A fully incremental model, that transforms raw web & mobile event data generated by the Snowplow JavaScript & mobile trackers into a series of derived tables of varying levels of aggregation.
Other
6 stars 5 forks source link

Real time modeling with new materialization options #41

Open leonard-henriquez opened 1 year ago

leonard-henriquez commented 1 year ago

Is your feature request related to a problem? Please describe.

It's hard to have always update to date data with low latency (<1 minute) with this package. This package doesn't provide a solution or documentation how to achieve this.

Describe the solution you'd like

Would it be possible to add support for other materialization option in the snowplow-web dbt package to achieve real-time transformation ? In Snowflake Dynamic tables could provide a more efficient way to maintain near-real-time data, avoiding the need for frequently running the models.

Additional context

Data Warehouse: Snowflake (but similar concepts exist in other data warehouse)

Are you interested in contributing towards this feature?

While I'm relatively new to dbt, I'd be open to contributing towards this feature with some guidance to get started.

rlh1994 commented 1 year ago

For anyone landing here - here is a thread on the issue: https://discourse.snowplow.io/t/data-modeling-in-real-time/8978

This is something we will likely look into but it's not currently on the immediate road map.

rlh1994 commented 1 year ago

Also linked to this issue https://github.com/snowplow/dbt-snowplow-utils/issues/136

miike commented 5 months ago

Moving this issue to the unified repo as the web model is in maintenance mode.