search

linkedin / cruise-control

Cruise-control is the first of its kind to fully automate the dynamic workload rebalance and self-healing of a Kafka cluster. It provides great value to Kafka users by simplifying the operation of Kafka clusters.
https://github.com/linkedin/cruise-control/tags
BSD 2-Clause "Simplified" License
2.68k stars 574 forks source link

java.lang.NoSuchMethodError: 'java.lang.String kafka.server.KafkaConfig.ListenersProp()' when running MetricsReporter for Kafka 3.8 #2168

Open HenryCaiHaiying opened 2 weeks ago

HenryCaiHaiying commented 2 weeks ago

When running embedded MetricsReporter with the upcoming Kafka 3.8, hit the following exception:

[2024-06-18 16:08:06,722] ERROR [KafkaServer id=1025] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) java.lang.NoSuchMethodError: 'java.lang.String kafka.server.KafkaConfig.ListenersProp()' at com.linkedin.kafka.cruisecontrol.metricsreporter.CruiseControlMetricsReporter.getBootstrapServers(CruiseControlMetricsReporter.java:129)

HenryCaiHaiying commented 2 weeks ago

The root cause is Kafka 3.8 (maybe also 3.7) refactored the core module for many configuration parameters, see this PR: https://github.com/apache/kafka/commit/1b301b30207ed8fca9f0aea5cf940b0353a1abca?diff=split&w=0#diff-8eb3e01716508551f203b0b37c2d5f951e93cce7ffed7c00c2b33633d7c8ed23

HenryCaiHaiying commented 2 weeks ago

The easiest fix is just to use hard-coded literal string 'listeners' for this config parameter, the name of this parameter is unlikely to change (otherwise it will break million's customer's server.properties). Although you can try to change the variable name reference from KafkaConfig.ListenersProp() to SocketServerConfigs.LISTENERS_CONFIG, but SocketServerConfigs is also a new class introduced in Kafka 3.8, to do that you would have to create a new CC branch migrate_to_3_8 to make the compilation work, but maintaining different branches for different kafka version is cumbersome and hard to maintain in the long run.

HenryCaiHaiying commented 2 weeks ago

See fix: https://github.com/linkedin/cruise-control/pull/2169/files