search

eventuate-foundation / eventuate-messaging-kafka

Other
15 stars 11 forks source link

Upgrade Kafka Client #58

Open cer opened 2 years ago

cer commented 2 years ago

The 2.3.0 client is very old. It should be replaced with 3.x

https://github.com/eventuate-foundation/eventuate-messaging-kafka/blob/32951a195388f047090ebc9776a2fe005fe81fea/eventuate-messaging-kafka-common/build.gradle#L5

We should consider upgrading the broker too:

https://github.com/eventuate-foundation/eventuate-messaging-kafka/blob/389ee45be761d61ccfda628a70d3a9d4f2594d59/kafka/Dockerfile#L2

ddaeyongee commented 2 years ago

@cer Hi. This is Taeyong from LG organization.

Currently, In our project, we do not use the apache kafka client license directly, but purchase a paid license for the confluent platform and use it. The confluent platform version used is 6.1, and the confluent(kafka) version 6.1.1 is compatible with apache kafka 2.7.x. According to the "Confluent Platform 3.2 and later" section of confluent's guide,

 "Kafka Java Clients that are included in Confluent Platform 3.2 (Kafka version 0.10.2) and later are compatible with any Kafka broker that is included in Confluent Platform 3.0 and later. "

link : https://docs.confluent.io/platform/current/installation/versions-interoperability.html#ak-java-clients

So, compatibility between kafka client 3.x and confluent(kafka) 6.1.1 seems to be fine. Thank you for your active support.

confluent_compatible_version

cer commented 2 years ago

Also, need to review/optimize:

For example, enable idempotent producer + ack=all + etc.

cer commented 2 years ago

Notes on version compatibility:

cer commented 2 years ago

Kafka Client 3.1.1 seems like a reasonable choice - used by Spring Boot dependencies.