search

ClickHouse / clickhouse-go

Golang driver for ClickHouse
Apache License 2.0
2.88k stars 553 forks source link

Is batch.Flush() Atomic? #903

Open kuro337 opened 1 year ago

kuro337 commented 1 year ago

I have a Table in Clickhouse as a ReplicatedMergeTree.

I am inserting large parquet files and I want to confirm whether Batch.flush() offers Atomicity as a guarantee?

Is it possible flushing will only insert rows partially?

Thank you for any assistance , relevant answers will help a lot!

jkaflik commented 1 year ago

hey @Chinmay337

from a protocol view, there is a such guarantee. I'm not sure what it looks like later in ClickHouse itself with ReplicatedMergeTree. Will ask around and get back with an answer to you.

Lastin commented 5 months ago

I am not 100% sure, but I've observed that when using batch.Flush() on table using ReplicatedMergeTree as the engine, my data wasn't being inserted altogether into my cluster (small cluster of 4 shards). There was no errors in the client indicating the issue. Using batch.Send() followed by creating a new batch each time has resolved the issue.