search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
95 stars 4.92k forks source link

Update Kafka dependencies to prepare for KIP-896 #41165

Open nhaq-confluent opened 1 month ago

nhaq-confluent commented 1 month ago

Hello!

I am from Confluent, an organization contributing to Apache Kafka, and wanted to raise a request in regards to KIP-896.

In short, KIP-896 removes compatibility of certain client protocol API versions in Apache Kafka(AK) 4.0, being targeted for around January 2025. Since there is a dependency on Kafka in beats (in particular to a forked version of the Sarama client) I wanted to raise awareness that there could be some changes necessary to ensure a smooth experience for those using beats and Kafka. Here are the recommendations.

  1. The latest versions of beats (7.X and higher) appear to be using sarama 1.29.1, which is good enough from the version perspective, however the Kafka version would need to be set to 2.1.0 manually by users to make this work. Recommend updating the default to be 2.1.0 or higher.

  2. Recommend updating the sarama dependency to the latest version possible in future beats versions

  3. Updating the beats docs, advising users of this situation and to set the Kafka version config to 2.1.0

Thank you!

ijuma commented 1 month ago

Reference to the PR where the default for Sarama was changed: https://github.com/IBM/sarama/pull/2574

nhaq-confluent commented 2 weeks ago

@ruflin @andrewkroh could you please comment on this issue?

elasticmachine commented 2 weeks ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

pierrehilbert commented 2 weeks ago

Hello @nhaq-confluent Thanks for raising this issue and bringing some context to it. We are currently discussing about what would be the best timing to change the default version to 2.1.0 as suggested.

ijuma commented 6 days ago

Thanks. One important detail is that 2.1.0 is truly ancient at this point, it was released 6 years ago (in Nov 2018). So, we should consider changing the default for older releases too to avoid the situation where they are completely broken with Apache Kafka 4.0 (due in 2-3 months).