housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
516 stars 118 forks source link

运行时出现kafka消费异常 #49

Closed ALEXuH closed 3 years ago

ALEXuH commented 4 years ago

初始化完成开始消费时出现此错误,用的1.5的版本,请问该如何解决呢 2020/04/26 15:10:19 Initial [clickhouse_sinker] &creator.Config{ Kafka: { "kfk": &creator.KafkaConfig{ Brokers: "127.0.0.1:19091,127.0.0.1:19092,127.0.0.1:19093", Sasl: struct { Password string; Username string }{ Password: "", Username: "", }, Version: "0.10.2.0", }, }, Clickhouse: { "ch": &creator.ClickHouseConfig{ Db: "ailpha", Host: "127.0.0.1", Port: 9101, Username: "default", Password: "", MaxLifeTime: 300, DnsLoop: false, }, }, Tasks: []*creator.Task{ &creator.Task{ Name: "request", Kafka: "kfk", Topic: "daily", ConsumerGroup: "testgroup", Earliest: true, Parser: "json", CsvFormat: []string{}, Delimiter: "", Clickhouse: "ch", TableName: "daily", AutoSchema: true, ExcludeColumns: []string{}, Dims: []struct { Name string; Type string }{}, Metrics: []struct { Name string; Type string }{}, FlushInterval: 0, BufferSize: 90000, }, }, Common: struct { FlushInterval int; BufferSize int; LogLevel string }{ FlushInterval: 5, BufferSize: 90000, LogLevel: "debug", }, } 2020/04/26 15:10:19 [I] Prepare sql=> INSERT INTO ailpha.daily () VALUES () 2020/04/26 15:10:19 Initial [clickhouse_sinker] complete 2020/04/26 15:10:19 [I] start to dial kafka 127.0.0.1:19091,127.0.0.1:19092,127.0.0.1:19093 2020/04/26 15:10:19 [E] Error from consumer: EOF %!v(MISSING) 2020/04/26 15:10:19 [E] Error from consumer: EOF %!v(MISSING) 2020/04/26 15:10:19 [E] Error from consumer: EOF %!v(MISSING) 2020/04/26 15:10:19 [E] Error from consumer: EOF %!v(MISSING) ...

sundy-li commented 4 years ago

目测是kafka版本有问题, 你改为0.10.2 试试

yuzhichang commented 3 years ago

@ALEXuH The master branch has replaced sarama with kafka-go.Please have a try.

yuzhichang commented 3 years ago

@ALEXuH I'll close the issue. Please leave a comment if master doesn't fix the problem.