ClickHouse / dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Apache License 2.0
255 stars 113 forks source link

LowCardinality/Nullable columns are incompatible with column constraints #274

Closed rjoelnorgren closed 6 months ago

rjoelnorgren commented 7 months ago

Describe the bug

Specifying a LowCardinality or Nullable type for a column results in a compilation error due to model contract failure.

Steps to reproduce

  1. Create a model with an an enforced contract
  2. Specify a column data_type as Nullable or LowCardinality

Expected behaviour

We can specify columns as Nullable or LowCardinality.

Code examples, such as models or profile settings

schema.yml

models:
  - name: nested_contract_model
    config:
      contract:
        enforced: true
      materialized: table
    columns:
      - name: low_card_column
        data_type: LowCardinality(String)
      - name: nullable_column
        data_type: Nullable(String)

low_card_model.sql

select toLowCardinality('hello') as low_card_column, null::Nullable(String) as nullable_column

dbt and/or ClickHouse server logs

dbt run -s nested_contract_model
20:32:45  Running with dbt=1.7.13
20:32:45  Registered adapter: clickhouse=1.7.6
20:32:45  Unable to do partial parsing because saved manifest not found. Starting full parse.
20:32:45  Found 3 models, 1 seed, 1 operation, 1 test, 3 sources, 0 exposures, 0 metrics, 431 macros, 0 groups, 0 semantic models
20:32:45  
20:32:46  
20:32:46  Running 1 on-run-start hook
20:32:46  1 of 1 START hook: jaffle_shop.on-run-start.0 .................................. [RUN]
20:32:46  1 of 1 OK hook: jaffle_shop.on-run-start.0 ..................................... [OK in 0.00s]
20:32:46  
20:32:46  Concurrency: 4 threads (target='dev')
20:32:46  
20:32:46  1 of 1 START sql table model `jaffle_shop`.`nested_contract_model` ............. [RUN]
20:32:46  1 of 1 ERROR creating sql table model `jaffle_shop`.`nested_contract_model` .... [ERROR in 0.03s]
20:32:46  
20:32:46  Finished running 1 table model, 1 hook in 0 hours 0 minutes and 0.18 seconds (0.18s).
20:32:46  
20:32:46  Completed with 1 error and 0 warnings:
20:32:46  
20:32:46    Compilation Error in model nested_contract_model (models/jaffle_shop/nested_contract_model.sql)
  This model has an enforced contract that failed.
  Please ensure the name, data_type, and number of columns in your contract match the columns in your model's definition.

  | column_name     | definition_type | contract_type          | mismatch_reason    |
  | --------------- | --------------- | ---------------------- | ------------------ |
  | low_card_column | String          | LowCardinality(String) | data type mismatch |
  | nullable_column | String          | Nullable(String)       | data type mismatch |

  > in macro clickhouse__get_assert_columns_equivalent (macros/column_spec_ddl.sql)
  > called by macro get_assert_columns_equivalent (macros/relations/column/columns_spec_ddl.sql)
  > called by macro create_table_or_empty (macros/materializations/table.sql)
  > called by macro clickhouse__create_table_as (macros/materializations/table.sql)
  > called by macro create_table_as (macros/relations/table/create.sql)
  > called by macro default__get_create_table_as_sql (macros/relations/table/create.sql)
  > called by macro get_create_table_as_sql (macros/relations/table/create.sql)
  > called by macro statement (macros/etc/statement.sql)
  > called by macro materialization_table_clickhouse (macros/materializations/table.sql)
  > called by model nested_contract_model (models/jaffle_shop/nested_contract_model.sql)
20:32:46  
20:32:46  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
rjoelnorgren commented 7 months ago

A few way this could be resolved, but the underlying issues is related to correctly preserving the nesting of types (https://github.com/ClickHouse/dbt-clickhouse/blob/main/dbt/adapters/clickhouse/column.py#L123-L127). This is also important for #3 which would need to support cases such as AggregateFunction(uniq, Nullable(Int32)).