StarRocks / dbt-starrocks

dbt-starrocks contains all of the code enabling dbt to work with StarRocks
10 stars 4 forks source link

#4 Table supports multiple data models #10

Closed motto1314 closed 2 years ago

motto1314 commented 2 years ago

4

  1. Table supports selecting one from Duplicate Key/Unique Key/Primary Key
  2. Table supports set ENGINE
  3. Table supports set keys
  4. Table supports set DISTRIBUTED BY
  5. Table supports set PROPERTIES
  6. Table supports set PARTITION BY

Notice:

  1. Create table as can only set engine='OLAP' and table_type='DUPLICATE'
  2. distributed_by is must

Test:

test_dbt_empty: empty test_dbt_base: base [ERROR: distributed_by is must ] test_dbt_ephemeral: ephemeral [ERROR: not support with ] test_dbt_incremental: incremental [ERROR: distributed_by is must ] test_dbt_snapshot_strategy_timestamp: snapshot_strategy_timestamp [ERROR: not support with ] test_dbt_snapshot_strategy_check_cols: snapshot_strategy_check_cols [ERROR: not support with ] test_dbt_data_test: data_test test_dbt_schema_test: schema_test [ERROR: distributed_by is must ] test_dbt_ephemeral_data_tests: data_test_ephemeral_models [ERROR: not support with ]

example:

dbt seed properties(whatever_you_want.yml):

Minimum configuration:

config:
  distributed_by: ['id']

Complete configuration:

config:
  engine: 'OLAP'
  keys: ['id', 'name', 'some_date']
  table_type: 'PRIMARY'     //PRIMARY or DUPLICATE or UNIQUE
  distributed_by: ['id']
  buckets: 3                //default 10
  partition_by: ['some_date']
  partition_by_init: ["PARTITION p1 VALUES [('1971-01-01 00:00:00'), ('1991-01-01 00:00:00')),PARTITION p1972 VALUES [('1991-01-01 00:00:00'), ('1999-01-01 00:00:00'))"]
  properties: {"replication_num":"1", "in_memory": "true"}

dbt run config(table/incremental):

Minimum configuration:

{{ config(materialized=var("materialized_var", "table"), distributed_by=['id'])}}
{{ config(materialized='incremental', distributed_by=['id']) }}

Complete configuration:

{{ config(materialized='table', engine='OLAP', buckets=32, distributed_by=['id'], properties={"in_memory": "true"}) }}
{{ config(materialized='incremental', engine='OLAP', buckets=32, distributed_by=['id'], properties={"in_memory": "true"}) }}