I know this has been discussed before but it is still an issue for us...
Frequent failures of incremental (delete+insert) models in clustered clickhouse (24.2.2.71). Re-running several times normally completes the process, however this seems to occur frequently enough to cause us difficulties.
The error is raised on the following step :
delete from
`dbt_prod_marts`.`mrt_my_table`
where
(
event_date, x,y,z
) in (
select
event_date,
x,y,z
from
`dbt_prod_marts`.`mrt_my_table__dbt_new_data_751e553e_44e5_4a12_99b7_fa79f323561a`
)
)
I think that perhaps what might be happening is that the 'temporary' table mrt_my_table__dbt_new_data_751e553e_44e5_4a12_99b7_fa79f323561a is not fully replicated by the time the deletion is run.
Is there a way that we can wait for replication of the 'temporary' table to complete before performing the delete step?
Steps to reproduce
create cluster with 2 replica nodes
create delete+insert incremental model as a ReplicatedMergetree
Materialize the table (may need a significant amount of data and possibly multiple attempts to reproduce).
Expected behaviour
Should robustly materialize the table
Code examples, such as models or profile settings
I have allow_nondeterministic_mutations set in my user profile, but the problem still occurs.
Describe the bug
I know this has been discussed before but it is still an issue for us...
Frequent failures of incremental (delete+insert) models in clustered clickhouse (24.2.2.71). Re-running several times normally completes the process, however this seems to occur frequently enough to cause us difficulties.
The error is raised on the following step :
I think that perhaps what might be happening is that the 'temporary' table
mrt_my_table__dbt_new_data_751e553e_44e5_4a12_99b7_fa79f323561a
is not fully replicated by the time the deletion is run.Is there a way that we can wait for replication of the 'temporary' table to complete before performing the delete step?
Steps to reproduce
Expected behaviour
Should robustly materialize the table
Code examples, such as models or profile settings
I have
allow_nondeterministic_mutations
set in my user profile, but the problem still occurs.dbt and/or ClickHouse server logs
Configuration
Environment
ClickHouse server