search

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse
https://housepower.github.io/clickhouse_sinker
Apache License 2.0
515 stars 118 forks source link

commit offset fails during stopping task #142

Closed yuzhichang closed 1 year ago

yuzhichang commented 2 years ago 
{"level":"warn","ts":"2022-01-20T11:31:08.029+0800","msg":"unable to commit offset for topic partition","group":"metric-010","topic":"perf-metric","partition":2,"commit_from":29746158,"commit_to":29747354,"commit_epoch":-1,"error_code":25}
{"level":"debug","ts":"2022-01-20T11:31:08.029+0800","msg":"updated committed","group":"metric-010","to":"perf-metric[]"}
{"level":"debug","ts":"2022-01-20T11:31:08.029+0800","msg":"left CommitOffsetsSync","group":"metric-010"}
{"level":"fatal","ts":"2022-01-20T11:31:08.029+0800","msg":"Batch.Commit failed with permanent error","task":"sink-metric","error":"UNKNOWN_MEMBER_ID: The coordinator is not aware of this member.","stacktrace":"github.com/housepower/clickhouse_sinker/output.(*ClickHouse).loopWrite\n\t/data01/zhichyu/clickhouse_sinker/output/clickhouse.go:178\ngithub.com/housepower/clickhouse_sinker/output.(*ClickHouse).Send.func1\n\t/data01/zhichyu/clickhouse_sinker/output/clickhouse.go:99\ngithub.com/housepower/clickhouse_sinker/util.(*WorkerPool).wokerFunc\n\t/data01/zhichyu/clickhouse_sinker/util/workerpool.go:59"}

sinker_metric.log

yuzhichang commented 2 years ago

another failure log sinker_metric.log

yuzhichang commented 1 year ago

Newer revision of clickhouse_sinker will retry on such error for limited times.