search

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
518 stars 118 forks source link

investigate franz-go, another kafka go client sdk #128

Closed yuzhichang closed 2 years ago

yuzhichang commented 3 years ago

https://github.com/twmb/franz-go is used by https://github.com/cloudhut/kminion. Looks good.

yuzhichang commented 2 years ago

Benchmark shows franz-go is the winner.

eoi@vm116:/data01/zhichyu/franz-go/examples/bench$ go run . --brokers 192.168.102.114:9092,192.168.102.115:9092,192.168.102.116:9092 --topic gaoju_test --consume
258.93 MiB/s; 1227.41k records/s
469.74 MiB/s; 2224.66k records/s
469.90 MiB/s; 2225.46k records/s
364.54 MiB/s; 1727.14k records/s
470.07 MiB/s; 2226.37k records/s
eoi@vm116:/data01/zhichyu/franz-go/examples/bench/compare/sarama$ go run . --brokers 192.168.102.114:9092,192.168.102.115:9092,192.168.102.116:9092 --topic gaoju_test --consume
268.19 MiB/s; 1269.77k records/s
258.58 MiB/s; 1224.43k records/s
264.12 MiB/s; 1250.69k records/s
eoi@vm116:/data01/zhichyu/franz-go/examples/bench/compare/confluent-kafka-go$ go run . --brokers 192.168.102.114:9092,192.168.102.115:9092,192.168.102.116:9092 --topic gaoju_test --consume --group abcdefg
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
0.00 MiB/s; 0.00k records/s
41.36 MiB/s; 195.68k records/s
41.39 MiB/s; 195.77k records/s
41.34 MiB/s; 195.58k records/s
41.39 MiB/s; 195.78k records/s
eoi@vm116:/data01/zhichyu/franz-go/examples/bench/compare/segment$ go run . --brokers 192.168.102.114:9092,192.168.102.115:9092,192.168.102.116:90
92 --topic gaoju_test --consume
61.43 MiB/s; 287.89k records/s
57.90 MiB/s; 271.35k records/s
64.85 MiB/s; 303.93k records/s
64.56 MiB/s; 302.58k records/s
yuzhichang commented 2 years ago

master tip 7a23ba3a4b8ffd9239e7da0ddeaef5b031769006 changed default kafka client from sarama to franz.