arnoN7 / dbt-incremental-stream

DBT Package reproducing dbt incremental materialization leveraging on Snowflake streams
MIT License
25 stars 4 forks source link

Ability to handle dbt variables to filter out stream (on filtered views) #4

Open ramandatascientist opened 1 year ago

ramandatascientist commented 1 year ago

I would like to request a functionality that will allow user to define project variables (https://docs.getdbt.com/docs/build/project-variables) to filter set of rows from the source table without impacting other rows

example

lets say the underlying source contains clients info, so each row will represent a particular client. now we have a dbt model that looks like this

select * from {{}} where client_id= '{{ var("client_id") }}'

We like to perform now dbt run --vars {{client_id='abc'}} so that this job can only capture client ABC records from the source table and not all others

SQL behind the scene --> select * from client where client_id='abc'

since streams are one-time process only, so we like to add this if-else logic to handle above use case