housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
519 stars 118 forks source link

Sharding policy #83

Closed yuzhichang closed 4 years ago

yuzhichang commented 4 years ago

Custom sharding policy (by config shardingKey and shardingPolicy). By default, shard with (msg.Offset/batchSize)%ckNum. shardingKey is a column name. shardingPolicy is stripe,<size> or hash. The hash function used internally is xxHash64.