housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
514 stars 118 forks source link

没有平衡的写入数据 #162

Closed shouziws closed 1 year ago

shouziws commented 1 year ago

{ "clickhouse": { "hosts": [ [ "10.242.44.95" ], [ "10.242.168.107" ] ], "port": 9000, "db": "default", "username": "default", "password": "root", "retryTimes": 0 }, "kafka": { "brokers": "192.168.1.4:9092", "version": "2.5.0" }, "task": { "name": "test_auto_schema", "topic": "topic1", "consumerGroup": "test_auto_schema", "earliest": true, "parser": "json", "autoSchema": true, "tableName": "test_auto_schema", "excludeColumns": [ "day" ], "bufferSize": 50000 }, "logLevel": "info" }

上面是我的sinker配置,我手动测试写kafka,他只会向一个host clickhouse服务写数据,没有做到平衡的写数据

yuzhichang commented 1 year ago

sinker可以平衡写,batch级别的。你kafka里数据不够两个batch,看不出来效果

shouziws commented 1 year ago

如作者所说,是batch级别的,是我数据量不够多看不出效果,已测试是平衡写入的,已解决