ClickHouse / dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Apache License 2.0
255 stars 113 forks source link

Cannot change between view and distributed table on cluster #345

Open krisnej opened 3 months ago

krisnej commented 3 months ago

Describe the bug

We want to use dbt on a Clickhouse cluster with two nodes. As far as I can tell, you cannot switch from a view materialization to a distributed table, or vice versa.

Steps to reproduce

profiles.yml is set up with on_cluster.

Model materialized as view:

{{  
    config(  
        materialized='view',  
    )  
}}  

SELECT 1 as id

Run dbt build, successfully creates the view. Switch model to distributed_table:

{{  
    config(  
    materialized='distributed_table',
        order_by='id',
        sharding_key='cityHash64(id)',
        engine='ReplacingMergeTree'
    )  
}}  

SELECT 1 as id

Run dbt build again, this fails:

 Database Error in model test (models/test.sql)
  :HTTPDriver for [hostname]:[port] returned response code 404)
   Code: 60. DB::Exception: Table [schema].`test_local` doesn't exist. (UNKNOWN_TABLE) (version 24.1.8.22 (official build))

Also a run with --full-refresh does not help.

If you start with a clean distributed table (distributed + local table) and change the materialization to a view, it succeeds, by changing the test table to a view, but the test_local table remains.

Expected behaviour

Changing materialization deletes and creates the expected tables/view.

Configuration

Environment

ClickHouse server

Is this behaviour intended or expected? If not, a fix would be appreciated, as our current workaround for this is to drop and recreate the whole schema.