We have a very large table in BigQuery and have created a filtered, smaller materialized table for analysts. Each day new data gets added to the large table and subsequently to the small view as well.
We are using the dataform CLI to run the models. However, even though we haven't changed the materialized view, every time we run the dataform project, the materialized view will be recreated and we have to process the whole data of the large table again.
We would expect that the materialized view keeps the previously processed data.
BigQuery Materialized View Issue
We have a very large table in BigQuery and have created a filtered, smaller materialized table for analysts. Each day new data gets added to the large table and subsequently to the small view as well.
We are using the dataform CLI to run the models. However, even though we haven't changed the materialized view, every time we run the dataform project, the materialized view will be recreated and we have to process the whole data of the large table again.
We would expect that the materialized view keeps the previously processed data.
This is how we create the materialized view:
Thanks for your help!