ClickHouse / dbt-clickhouse

The Clickhouse plugin for dbt (data build tool)
Apache License 2.0
243 stars 99 forks source link

ALTER UPDATE/ALTER DELETE statement with subquery may be nondeterministic #282

Open the4thamigo-uk opened 4 months ago

the4thamigo-uk commented 4 months ago

Describe the bug

I know this has been discussed before but it is still an issue for us...

Frequent failures of incremental (delete+insert) models in clustered clickhouse (24.2.2.71). Re-running several times normally completes the process, however this seems to occur frequently enough to cause us difficulties.

The error is raised on the following step :

delete from 
  `dbt_prod_marts`.`mrt_my_table` 
where 
  (
    event_date, x,y,z
  ) in (
    select 
      event_date, 
      x,y,z
    from 
      `dbt_prod_marts`.`mrt_my_table__dbt_new_data_751e553e_44e5_4a12_99b7_fa79f323561a`
  )
)

I think that perhaps what might be happening is that the 'temporary' table mrt_my_table__dbt_new_data_751e553e_44e5_4a12_99b7_fa79f323561a is not fully replicated by the time the deletion is run.

Is there a way that we can wait for replication of the 'temporary' table to complete before performing the delete step?

Steps to reproduce

  1. create cluster with 2 replica nodes
  2. create delete+insert incremental model as a ReplicatedMergetree
  3. Materialize the table (may need a significant amount of data and possibly multiple attempts to reproduce).

Expected behaviour

Should robustly materialize the table

Code examples, such as models or profile settings

I have allow_nondeterministic_mutations set in my user profile, but the problem still occurs.

dbt and/or ClickHouse server logs

<Error> executeQuery: Code: 36. DB::Exception: ALTER UPDATE/ALTER DELETE statement with subquery may be nondeterministic, see allow_nondeterministic_mutations setting. (BAD_ARGUMENTS) (version 24.2.2.71 (official build)) (from 10.26.0.61:4171) (in query: /* {"app": "dbt", "dbt_version": "1.7.14", "profile_name": "clickhouse", "target_name": "prod", "node_id": "redacted*/
--
8. DB::HTTPHandler::processQuery(DB::HTTPServerRequest&, DB::HTMLForm&, DB::HTTPServerResponse&, DB::HTTPHandler::Output&, std::optional<DB::CurrentThread::QueryScope>&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000012f672d5
9. DB::HTTPHandler::handleRequest(DB::HTTPServerRequest&, DB::HTTPServerResponse&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000012f6bc10
10. DB::HTTPServerConnection::run() @ 0x0000000012fe5ffa
11. Poco::Net::TCPServerConnection::start() @ 0x00000000158bc9b2
12. Poco::Net::TCPServerDispatcher::run() @ 0x00000000158bd7b1
13. Poco::PooledThread::run() @ 0x00000000159b6387
14. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000159b49bd
15. ? @ 0x00007deb343be609
16. ? @ 0x00007deb342e3353

Configuration

Environment

ClickHouse server

rightx2 commented 2 months ago

Got a same error with same reason......