StarRocks / dbt-starrocks

dbt-starrocks contains all of the code enabling dbt to work with StarRocks
10 stars 4 forks source link

[materialized_view] why do not use swap instead of drop then create #49

Open Mrzyxing opened 2 weeks ago

Mrzyxing commented 2 weeks ago

After changed materialized view(or called MV in rest of content), we will drop exist MV then create as follow code:

{% macro starrocks__get_replace_materialized_view_as_sql(relation, sql, existing_relation, backup_relation, intermediate_relation) %}
    {{ starrocks__get_drop_relation_sql(existing_relation) }}
    {{ get_create_materialized_view_as_sql(relation, sql) }}
{% endmacro %}

And I think there are better way to alter asynchronous MV because starrocks support atomic swap as this doc. I will fork as internal project to use swap but also want to konw more ideas of community.

nicor88 commented 2 weeks ago

I'm not using StarRocks (yet), but I'm watching this adapter really closely. What it's being raised in this issue it's really really relevant. If I plan to use a table/MV in my presentation layer (e.g. Tableau/Looker) I want to have it high available, therefore no downtime (means that the table/MV must exists always, it's fine to have stale data, but it's not fine to have the no table/MV even for few seconds, because can create some interruptions to the users using the presentation layer.

Atomic swap sounds indeed a possibility to achieve what describe above, it will be amazing to have such feature in this dbt adapter.