Avro Support - Githubissues

housepower / clickhouse_sinker

Easily load data from kafka to ClickHouse

https://housepower.github.io/clickhouse_sinker

Apache License 2.0

515 stars 118 forks source link

Avro Support #129

Closed allapospelova closed 2 years ago

allapospelova commented 2 years ago

Supports decoding single-object Avro messages commonly used with Kafka and Confluent Schema Registry

Hello, I had a task to deploy the kafka and integrate with our local clickhouse service. I have added some code for that. Thank you for your work and maybe my little update will be helpful for you too.

sundy-li commented 2 years ago

Why not make Avro a parser like json/csv?

allapospelova commented 2 years ago

Why not make Avro a parser like json/csv?

@sundy-li This is the more about support AvroConfluent with Schema Registry

sundy-li commented 2 years ago

This LGTM, cc @yuzhichang

allapospelova commented 2 years ago

What's benefit of Avro+Json? The performance may be poor.

@yuzhichang How bad can performance be Avro+Json ? And how can this be avoided ?

yuzhichang commented 2 years ago

What's benefit of Avro+Json? The performance may be poor.

@yuzhichang How bad can performance be Avro+Json ? And how can this be avoided ?

Don't introduce confluent-kafka-go at this PR since github.com/riferrei/srclient is independent of the kafka client SDK.
At parsing phase, add an Avro parser. Use msg.Value[1:5] as Avro Schema ID and decode msg.Value[5:] per the schema. The schema need be the 1-to-1 mapping of the ClickHouse table columns.