search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
109 stars 4.93k forks source link

Update Kafka dependencies to prepare for KIP-896 #41165

Open nhaq-confluent opened 1 month ago

nhaq-confluent commented 1 month ago

Hello!

I am from Confluent, an organization contributing to Apache Kafka, and wanted to raise a request in regards to KIP-896.

In short, KIP-896 removes compatibility of certain client protocol API versions in Apache Kafka(AK) 4.0, being targeted for around January 2025. Since there is a dependency on Kafka in beats (in particular to a forked version of the Sarama client) I wanted to raise awareness that there could be some changes necessary to ensure a smooth experience for those using beats and Kafka. Here are the recommendations.

  1. The latest versions of beats (7.X and higher) appear to be using sarama 1.29.1, which is good enough from the version perspective, however the Kafka version would need to be set to 2.1.0 manually by users to make this work. Recommend updating the default to be 2.1.0 or higher.

  2. Recommend updating the sarama dependency to the latest version possible in future beats versions

  3. Updating the beats docs, advising users of this situation and to set the Kafka version config to 2.1.0

Thank you!

ijuma commented 1 month ago

Reference to the PR where the default for Sarama was changed: https://github.com/IBM/sarama/pull/2574

nhaq-confluent commented 1 month ago

@ruflin @andrewkroh could you please comment on this issue?

elasticmachine commented 1 month ago

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

pierrehilbert commented 1 month ago

Hello @nhaq-confluent Thanks for raising this issue and bringing some context to it. We are currently discussing about what would be the best timing to change the default version to 2.1.0 as suggested.

ijuma commented 3 weeks ago

Thanks. One important detail is that 2.1.0 is truly ancient at this point, it was released 6 years ago (in Nov 2018). So, we should consider changing the default for older releases too to avoid the situation where they are completely broken with Apache Kafka 4.0 (due in 2-3 months).

khushijain21 commented 2 weeks ago

Hello @nhaq-confluent @ijuma, is it possible to have access to 4.0.0 SNAPSHOT version for testing? This would help us perform upgrades and test it locally in our labs. TIA