ContentSquare / chproxy

Open-Source ClickHouse http proxy and load balancer
https://www.chproxy.org/
MIT License
1.28k stars 259 forks source link

[QUESTION] why only one replica node in each shard have a high load? #412

Closed fankuneee closed 6 months ago

fankuneee commented 6 months ago

https://github.com/ClickHouse/ClickHouse/issues/61716#issue-2200405050

Why, when sending a select to a ClickHouse cluster with multiple replicas, does only one replica node in each shard have a high load, while the other replica nodes have a low load, causing the node with high load to be unavailable at times?

I have 8 ClickHouse nodes, 4 shards, and 2 replicas per shard. I'm using chproxy for load balancing across my cluster. However, the result is that in each shard, only one replica is overloaded while the other remains idle. Are there any configuration options or solutions to resolve this issue?

also the load_balancing option in my config.xml is round_robin,but it seems not working.

and I use Distributed Table Engine

clickhouse-server version: v22.10.6.3-stable chproxy: 1.20.1

mga-chka commented 6 months ago

can you try with a version >= 1.25 or < 1.20? Cf this bug fix, there was an issue on the load balancing logic, it might explain your problem.