search

dbt-labs / dbt-labs-experimental-features

dbt support for database features which are not yet supported natively in dbt-core
Apache License 2.0
147 stars 43 forks source link

Bug: Materialized view with auto-refresh will not redeploy when definition has changed #22

Closed Luttik closed 5 months ago

Luttik commented 2 years ago

Currently materialized views with auto-refresh won't rebuild when the definition of the DBT model has changed. It is good that DBT doesn't refresh the views when either the calculated view is unchanged, but it definitely should be deployed the definition was changed.

I see two solutions for this:

  1. Check if the definition of the materialized view changed on the user's machine
  2. Check if the calculated materialized view differs from the materialized view that is currently deployed

I think the latter is the more elegant one, but I don't know which are feasible given the options working within dbt macro's.

The fix could either be applied in the refresh_materialized_view methods which are dependent on the specific data warehouse or extending the elif statement in line 19 of default/materialized_view to include one of the checks mentioned above.

jtcohen6 commented 2 years ago

@Luttik Love that you're thinking about this! What do you think about leveraging dbt's state-based selection to accomplish this? Then this would look something like:

# "full refresh" all MVs that have changes
dbt -s state:modified,config.materialized:materialized_view --full-refresh

# partial refresh of MVs without changes
dbt -s config.materialized:materialized_view --exclude state:modified
Luttik commented 2 years ago

I'm not sure if I'm tracking completely. However, I think that this should just work automagically without having to specify anything in addition to the dbt run command, after a run you expect the live definitions to be in-sync with your dbt definitions.

Not refreshing data vs not syncing the model is very different in my eyes, the first will correct itself at some point while the latter does not.

dataders commented 5 months ago

hey @Luttik thanks for opening this issue! Even though MVs & DTs now live in Core, this limitation that you call out still existis. I'm closing this issue in favor of https://github.com/dbt-labs/dbt-adapters/issues/234 so that we can track the use case and priority over there moving forward