Support KIP 392 in Kafka 2.4

[bczifra]

KIP 392 was merged in the latest Kafka version (2.4).

The new version reportedly does not break previous functionality, but allows a consumer to read from a replica that is not the leader, but is in sync. This would be useful in cases where there is a highly available Kafka cluster between data centers or regions, where Logstash would then read from the closest dc / region instead of the leader. This saves a lot on network hops.

Also see https://www.confluent.io/blog/multi-region-data-replication .

[Duchadian]

Updating the issue on the request of @bczifra . To make full use of the follower fetching functionality, the client.rack parameter needs to be passed to the Kafka Consumer. Besides this, no additional configuration should be needed if the brokers are set up correctly.

In short, two things would need to be changed about the Logstash Kafka integration:

• update the org.apache.kafka:kafka-clients dependency to 2.4.0
• client.rack would need to be added as a param to lib/logstash/inputs/kafka.rb , no changes are needed for the output

I saw that @ppf2 copied the bump request to the Kafka integration and input / output repos. Should I copy this request to those repos, or would you prefer to handle that?

[ppf2]

Probably best to handle this as part of https://github.com/logstash-plugins/logstash-integration-kafka/issues/14 (And if we need to backport to 9.x series for the non-integration plugins, we can use the related links there to cross reference). I have pasted the details from this request to https://github.com/logstash-plugins/logstash-integration-kafka/issues/14 Thx!