search

ClickHouse / ClickHouse

ClickHouse® is a real-time analytics DBMS
https://clickhouse.com
Apache License 2.0
37.66k stars 6.91k forks source link

Support for upcoming Kafka 4.0 #63697

Open aiven-sal opened 6 months ago

aiven-sal commented 6 months ago

Kafka 4.0 will remove some older protocol API versions (KIP-896). CH seems to rely at least on one of the deprecated API (i.e. ListOffsets v0) via its own librdkafka. It should be possible and easy to patch the current version of the lib to not use the deprecated version, but there might be more stuff we didn't notice yet. I think that the best solution is to update librdkafka to v1.8.2. If no one has anything against it, I'll be happy to work on this myself.

melvynator commented 6 months ago

@antaljanosbenjamin @KochetovNicolai

Do you have any objections for it?

antaljanosbenjamin commented 6 months ago

I remember we tried to upgrade librdkafka, but we abandond that idea, though I am not sure why. Maybe because of the patches we have in librdkafka makes it hard to upgrade and there was no really good reason to do so. This seems a good reason, because as @aiven-sal mentioned, there might be other issues that we don't notice. In short I think it would be good to upgrade to newer librdkafka version, but I am not sure how much work it is.