Closed shouziws closed 1 year ago
{ "clickhouse": { "hosts": [ [ "10.242.44.95" ], [ "10.242.168.107" ] ], "port": 9000, "db": "default", "username": "default", "password": "root", "retryTimes": 0 }, "kafka": { "brokers": "192.168.1.4:9092", "version": "2.5.0" }, "task": { "name": "test_auto_schema", "topic": "topic1", "consumerGroup": "test_auto_schema", "earliest": true, "parser": "json", "autoSchema": true, "tableName": "test_auto_schema", "excludeColumns": [ "day" ], "bufferSize": 50000 }, "logLevel": "info" }
上面是我的sinker配置,我手动测试写kafka,他只会向一个host clickhouse服务写数据,没有做到平衡的写数据
sinker可以平衡写,batch级别的。你kafka里数据不够两个batch,看不出来效果
如作者所说,是batch级别的,是我数据量不够多看不出效果,已测试是平衡写入的,已解决
{ "clickhouse": { "hosts": [ [ "10.242.44.95" ], [ "10.242.168.107" ] ], "port": 9000, "db": "default", "username": "default", "password": "root", "retryTimes": 0 }, "kafka": { "brokers": "192.168.1.4:9092", "version": "2.5.0" }, "task": { "name": "test_auto_schema", "topic": "topic1", "consumerGroup": "test_auto_schema", "earliest": true, "parser": "json", "autoSchema": true, "tableName": "test_auto_schema", "excludeColumns": [ "day" ], "bufferSize": 50000 }, "logLevel": "info" }
上面是我的sinker配置,我手动测试写kafka,他只会向一个host clickhouse服务写数据,没有做到平衡的写数据