DBT Core is unable to update tables with the Replicated engine, as DBT creates an empty copy of the destination table and, after completing the update (in my case, with materialized='incremental' and incremental_strategy='delete+insert', inserts data from the copy into the destination table and then deletes the copy. However, this process ignores the fact that Clickhouse is unable to create a copy of a replicated table, resulting in an exception being thrown (Code: 253. DB::Exception: Replica /clickhouse/tables/db/table/replicas/replica_name already exists).
Although this was expected, I attempted to use the MergeTree engine for updating, since it does not require replicating the table with new data. However, unexpectedly, DBT begins to freeze randomly while performing the updates. Sometimes, it works fine, but at other times, it appears to be deadlocked for some reason, regardless of the number of threads I use.
Steps to reproduce
create Replicated table
run update with incremental strategy
create replicated table
switch model engine to not replicated
run update with incremental strategy
Expected behaviour
DBT Core works properly with replicated tables in ClickHouse, and DBT Core does not randomly freeze while updating replicated tables.
Code examples, such as models or profile settings
{{ config(
order_by='(id)',
engine="ReplicatedMergeTree('/clickhouse/tables/dbt/my_table', 'my_replica')",
materialized='incremental',
unique_key=['id'],
incremental_strategy='delete+insert',
on_schema_change='fail'
) }}
select * from source_table
{% if is_incremental() %}
where id not in (select id from {{ this }})
{% endif %}
dbt and/or ClickHouse server logs
14:39:47
14:39:47 Finished running 2 incremental models in 0 hours 0 minutes and 13.53 seconds (13.53s).
14:39:47
14:39:47 Completed with 2 errors and 0 warnings:
14:39:47
14:39:47 Database Error in model some_model (models/model_name/model.sql)
HTTPDriver for http://46.101.104.158:8123 received ClickHouse error code 253
Code: 253. DB::Exception: Replica /clickhouse/tables/dbt/table/replicas/replica1 already exists. (REPLICA_ALREADY_EXISTS) (version 24.8.2.3 (official build))
14:39:47
14:39:47 Database Error in model some_model (models/model_name/model2.sql)
HTTPDriver for http://46.101.104.158:8123 received ClickHouse error code 253
Describe the bug
DBT Core is unable to update tables with the Replicated engine, as DBT creates an empty copy of the destination table and, after completing the update (in my case, with
materialized='incremental' and incremental_strategy='delete+insert'
, inserts data from the copy into the destination table and then deletes the copy. However, this process ignores the fact that Clickhouse is unable to create a copy of a replicated table, resulting in an exception being thrown (Code: 253. DB::Exception: Replica /clickhouse/tables/db/table/replicas/replica_name already exists
).Although this was expected, I attempted to use the MergeTree engine for updating, since it does not require replicating the table with new data. However, unexpectedly, DBT begins to freeze randomly while performing the updates. Sometimes, it works fine, but at other times, it appears to be deadlocked for some reason, regardless of the number of threads I use.
Steps to reproduce
create Replicated table
run update with incremental strategy
create replicated table
switch model engine to not replicated
run update with incremental strategy
Expected behaviour
DBT Core works properly with replicated tables in ClickHouse, and DBT Core does not randomly freeze while updating replicated tables.
Code examples, such as models or profile settings
dbt and/or ClickHouse server logs
14:39:47
14:39:47 Finished running 2 incremental models in 0 hours 0 minutes and 13.53 seconds (13.53s). 14:39:47
14:39:47 Completed with 2 errors and 0 warnings: 14:39:47
14:39:47 Database Error in model some_model (models/model_name/model.sql) HTTPDriver for http://46.101.104.158:8123 received ClickHouse error code 253 Code: 253. DB::Exception: Replica /clickhouse/tables/dbt/table/replicas/replica1 already exists. (REPLICA_ALREADY_EXISTS) (version 24.8.2.3 (official build)) 14:39:47
14:39:47 Database Error in model some_model (models/model_name/model2.sql) HTTPDriver for http://46.101.104.158:8123 received ClickHouse error code 253
Configuration
{{ config( order_by='(id)', engine="ReplicatedMergeTree('/clickhouse/tables/dbt/my_table', 'my_replica')", materialized='incremental', unique_key=['id'], incremental_strategy='delete+insert', on_schema_change='fail' ) }}
Environment
ClickHouse server