andsel
commented
1 year ago The CI error that's happening on 7.x branch is:
Java::OrgApacheKafkaCommon::KafkaException:
Failed to construct kafka producer
# org.apache.kafka.clients.producer.KafkaProducer.<init>(org/apache/kafka/clients/producer/KafkaProducer.java:468)
# org.apache.kafka.clients.producer.KafkaProducer.<init>(org/apache/kafka/clients/producer/KafkaProducer.java:291)
# org.apache.kafka.clients.producer.KafkaProducer.<init>(org/apache/kafka/clients/producer/KafkaProducer.java:318)
# org.apache.kafka.clients.producer.KafkaProducer.<init>(org/apache/kafka/clients/producer/KafkaProducer.java:303)
# jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
# jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(jdk/internal/reflect/NativeConstructorAccessorImpl.java:62)
# jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(jdk/internal/reflect/DelegatingConstructorAccessorImpl.java:45)
# java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:490)
# usr.share.plugins.plugin.spec.integration.inputs.kafka_spec.write_some_data_to(/usr/share/plugins/plugin/spec/integration/inputs/kafka_spec.rb:404)
# usr.share.plugins.plugin.spec.integration.inputs.kafka_spec./usr/share/plugins/plugin/spec/integration/inputs/kafka_spec.rb(/usr/share/plugins/plugin/spec/integration/inputs/kafka_spec.rb:411)
# java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:710)
# ------------------
# --- Caused by: ---
# Java::JavaLang::ClassNotFoundException:
# com.fasterxml.jackson.databind.json.JsonMapper
# java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)happens also with 8.x up to 8.2.3
With LS 8.3.0 jrjackson switched from 0.4.14 to 0.4.15 and this makes the jackson JSON library to switch from 2.9.10 to 2.13.3.
Now, the class that can't be found com.fasterxml.jackson.databind.json.JsonMapper is present in Jackson library only starting from 2.10 .
Kafka Client switched jackson-databind
- from
2.9.9.3to2.10.0with https://github.com/apache/kafka/pull/7411 shipped in2.x - from
2.10.xto2.12.3with https://github.com/apache/kafka/pull/10778 which is part of3.0
Release notes
Update kafka client to 3.3.1, requires Logstash >= 8.3.0. Deprecated
defaultvalue for settingclient_dns_lookup, when used it's forced touse_all_dns_ipsWhat does this PR do?
Updated the Kafka client library used from
2.8.1to3.3.1and Schema Registry from6.2.2to7.3.0.This PR updates the client library and, due to the removal of
zookeeperflag in launching integration Kafka instance, update also the bash scripts used in integration testing. The update of the Schema Registry forces the dependency on Logstash 8.3 because it's the first that ships Jackson2.13.3which is strictly required during the instantiation of the client.Why is it important/What is the impact to the user?
Updating the client requires a library (Jackson) that's available from Logstash
8.3.0so this is a requirement starting from this version of the plugin. The new Kafka client dropped thedefaultvalue for configurationclient_dns_lookup. Starting from Kafka2.6.0it had been deprecated and was removed in Kafka3.0. The default value for plugin settingclient_dns_lookupswitched fromdefaulttouse_all_dns_ipswhich if a user doesn't customize i,t doesn't create any problem. If a user explicitly setclient_dns_lookuptodefaultand updates the plugin to this version then the value is forced touse_all_dns_ips. Simple explanation whydefaultis not a good choice, see here.Checklist
[ ] I have made corresponding change to the default configuration files (and/or docker env variables)[ ] I have added tests that prove my fix is effective or that my feature worksAuthor's Checklist
How to test this PR locally
cd-ing into the uncompressed folderlogstash_integration_topic_plainto be later used into the Logstash pipelineGemfileoutput { stdout { codec => rubydebug {metadata => true } } }
The events has to be received and logged by Logstash.
Shutdown and cleanup:
Related issues
Use cases
As a Kafka user I want to be able to use a Logstash plugin aligned to the latest major version of Kafka.
Screenshots
Logs