search

StarRocks / dbt-starrocks

dbt-starrocks contains all of the code enabling dbt to work with StarRocks
10 stars 4 forks source link

StarRocks DBT connector to support schema evolution #17

Open alberttwong opened 4 months ago

alberttwong commented 4 months ago

Does our StarRocks DBT connector handle the following DBT schema evolution changes? https://github.com/dbt-labs/dbt-spark/issues/124#issuecomment-729058371

Handle schema evolution with and without partitioning. 

* The full import, without partitioning. As @charlottevdscheun mentioned earlier in https://github.com/dbt-labs/dbt-spark/pull/117. Currently we use INSERT INTO, but I would suggest to replace this by CREATE OR REPLACE TABLE, this allows us to atomically update the table, but also allowing changing the schema. This will keep full history of the table, and is fully supported by Delta.
* In the case of partition by, forward compatible schema evolution is allowed. We can add new fields to partitions, and they will be just null for the other partitions.