Closed ilovechai closed 3 years ago
@wbittles is this something you could look at?
I've been able to pass additional properties to the kafka consumer/producer, for example for authentication/ssl related configuration as per https://github.com/planetf1/egeria/blob/master/open-metadata-implementation/adapters/open-connectors/event-bus-connectors/open-metadata-topic-connectors/kafka-open-metadata-topic-connector/README.md without any changes. I've not checked this for a few release but do not believe anything has changed
@planetf1 we are able to pass additional properties as well, but it's this property that's picked up but cannot find the Round Robin Partitioner class in kafka client package.
It's not obvious how I can help with this , is the kafka version being used 2.4+ ?
The error message is coming from Kafka - Egeria is just logging it (badly - raised issue #5028). This means that Kafka is picking up the properties - it just can not use it.
Are these classes on the classpath?
It seems org.apache.kafka.clients.producer.RoundRobinPartitioner was added in Kafka 2.4
@mandy-chessell egeria 2.4 uses kafka-client:2.6.0
https://github.com/odpi/egeria/blob/egeria-release-2.4/build.gradle#L122. We also use kafka-client:2.3.1
, not sure if that's overriding the transitive dependecy that egeria has. I'll upgrade kafka-client
, test the changes, and get back.
okay looks like upgrading to kafka-client:2.7
picks up the round robin partitioner class.
Wed Apr 07 23:59:12 GMT 2021 igc_omrs Startup OMAG-ADMIN-0004 The igc_omrs server has successfully completed start up. The following services are running: [Open Metadata Repository Services (OMRS)]
2021-04-07 23:59:12.458 INFO [ pool-4-thread-1] o.a.k.c.p.ProducerConfig : ProducerConfig values:
acks = -1
batch.size = 16384
bootstrap.servers = [kafka:9093]
buffer.memory = 33554432
client.dns.lookup = use_all_dns_ips
..
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.RoundRobinPartitioner <--
Before offset:
[root###]# kubectl exec -it kafka-0 -- /opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic open-metadata.repository-services.cohort.one_catalog.OMRSTopic
open-metadata.repository-services.cohort.one_catalog.OMRSTopic:0:7649
open-metadata.repository-services.cohort.one_catalog.OMRSTopic:1:9307
open-metadata.repository-services.cohort.one_catalog.OMRSTopic:2:0
After offset:
[root###]# kubectl exec -it kafka-0 -- /opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic open-metadata.repository-services.cohort.one_catalog.OMRSTopic
open-metadata.repository-services.cohort.one_catalog.OMRSTopic:0:7700 <---
open-metadata.repository-services.cohort.one_catalog.OMRSTopic:1:9358 <---
open-metadata.repository-services.cohort.one_catalog.OMRSTopic:2:52 <---
Property that worked:
"partitioner.class": "org.apache.kafka.clients.producer.RoundRobinPartitioner"
Looks like we can close this?
We are using egeria-2.4 version at the moment, and trying to change the
partitioner.class
property in eventbus config, but kafka producer is unable to pick that up.Property key and values tried:
Below is omag log error:
Omag server config
Does kafka client in egeria-2.4 pick up/use the
RoundRobin
partitioner.class?