search

rollno748 / di-kafkameter

JMeter Plugin to load test Apache Kafka topics/brokers
34 stars 11 forks source link

java.lang.NoSuchMethodError on running kafka consumer sampler #14

Closed hengkybuntoroalldata closed 1 year ago

hengkybuntoroalldata commented 1 year ago

Hello, I'm getting this error when running the consumer sampler

java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.poll(Ljava/time/Duration;)Lorg/apache/kafka/clients/consumer/ConsumerRecords;
    at com.di.jmeter.kafka.sampler.KafkaConsumerSampler.getConsumerRecords(KafkaConsumerSampler.java:90) ~[di-kafkameter-1.2.jar:?]
    at com.di.jmeter.kafka.sampler.KafkaConsumerSampler.sample(KafkaConsumerSampler.java:73) ~[di-kafkameter-1.2.jar:?]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) ~[ApacheJMeter_core.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) ~[ApacheJMeter_core.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) ~[ApacheJMeter_core.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) ~[ApacheJMeter_core.jar:5.5]
    at java.lang.Thread.run(Thread.java:834) ~[?:?]
2023-03-29 16:37:42,761 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2023-03-29 16:37:42,765 INFO c.d.j.k.c.KafkaConsumerConfig: Kafka consumer client connection terminated
2023-03-29 16:37:42,765 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, *local*)

The Config I'm using is image and the sampler configuration is image

I'm using di-kafkameter-1.2.jar and the jmeter version is 5.5 any insight on this error?

Thank you

rollno748 commented 1 year ago

Hello @hengkybuntoroalldata

It's not the issue with the plugin, by looking at the log -> the polling is getting timed out

QQ: Is your topic name right? it was given as di-kafkameter

hengkybuntoroalldata commented 1 year ago

Hello, thanks for the fast response I have tried to change the poll to 60000 like you suggest, but i still seeing the same error image

2023-03-29 17:27:39,527 INFO o.a.j.s.FileServer: Default base='C:\Users\hengk\Desktop\apache-jmeter-5.5\bin'
2023-03-29 17:27:39,531 INFO o.a.j.g.a.Load: Loading file: C:\Users\hengk\Desktop\apache-jmeter-5.5\dikafkameter.jmx
2023-03-29 17:27:39,532 INFO o.a.j.s.FileServer: Set new base='C:\Users\hengk\Desktop\apache-jmeter-5.5'
2023-03-29 17:27:39,798 INFO o.a.j.s.SaveService: Testplan (JMX) version: 2.2. Testlog (JTL) version: 2.2
2023-03-29 17:27:39,839 INFO o.a.j.s.SaveService: Using SaveService properties version 5.0
2023-03-29 17:27:39,842 INFO o.a.j.s.SaveService: Using SaveService properties file encoding UTF-8
2023-03-29 17:27:39,845 INFO o.a.j.s.SaveService: Loading file: C:\Users\hengk\Desktop\apache-jmeter-5.5\dikafkameter.jmx
2023-03-29 17:27:40,388 INFO o.a.j.s.FileServer: Set new base='C:\Users\hengk\Desktop\apache-jmeter-5.5'
2023-03-29 17:27:51,397 INFO o.a.j.e.StandardJMeterEngine: Running the test!
2023-03-29 17:27:51,399 INFO o.a.j.s.SampleEvent: List of sample_variables: []
2023-03-29 17:27:51,399 INFO o.a.j.s.SampleEvent: List of sample_variables: []
2023-03-29 17:27:51,401 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must contain the string: '.functions.'
2023-03-29 17:27:51,401 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must not contain the string: '.gui.'
2023-03-29 17:27:52,185 INFO o.a.j.t.TestPlan: added C:\Users\hengk\Desktop\apache-jmeter-5.5\kafka311\libs to classpath
2023-03-29 17:27:52,194 INFO c.d.j.k.c.KafkaConsumerConfig: Setting up Additional properties
2023-03-29 17:27:52,209 INFO o.a.k.c.c.ConsumerConfig: ConsumerConfig values: 
    auto.commit.interval.ms = 5000
    auto.offset.reset = earliest
    bootstrap.servers = [docker-kafka-1:19092, docker-kafka-2:29092, docker-kafka-3:39092, docker-kafka-4:49092]
    check.crcs = true
    client.id = 
    connections.max.idle.ms = 540000
    enable.auto.commit = true
    exclude.internal.topics = true
    fetch.max.bytes = 52428800
    fetch.max.wait.ms = 500
    fetch.min.bytes = 1
    group.id = consumer-group-1
    heartbeat.interval.ms = 3000
    interceptor.classes = []
    internal.leave.group.on.close = true
    isolation.level = read_uncommitted
    key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
    max.partition.fetch.bytes = 1048576
    max.poll.interval.ms = 300000
    max.poll.records = 10
    metadata.max.age.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
    receive.buffer.bytes = 65536
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 305000
    retry.backoff.ms = 100
    sasl.jaas.config = null
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.mechanism = GSSAPI
    security.protocol = PLAINTEXT
    send.buffer.bytes = 131072
    session.timeout.ms = 10000
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    ssl.endpoint.identification.algorithm = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLS
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    value.deserializer = class org.apache.kafka.common.serialization.StringDeserializer

2023-03-29 17:27:53,295 INFO o.a.k.c.u.AppInfoParser: Kafka version : 1.1.0
2023-03-29 17:27:53,295 INFO o.a.k.c.u.AppInfoParser: Kafka commitId : fdcf75ea326b8e07
2023-03-29 17:27:53,298 INFO c.d.j.k.c.KafkaConsumerConfig: Kafka consumer client successfully Initialized
2023-03-29 17:27:53,300 INFO o.a.j.g.u.JMeterMenuBar: setRunning(true, *local*)
2023-03-29 17:27:53,366 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : Thread Group
2023-03-29 17:27:53,366 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group Thread Group.
2023-03-29 17:27:53,367 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error
2023-03-29 17:27:53,367 INFO o.a.j.t.ThreadGroup: Starting thread group... number=1 threads=1 ramp-up=1 delayedStart=false
2023-03-29 17:27:53,372 INFO o.a.j.t.ThreadGroup: Started thread group number 1
2023-03-29 17:27:53,373 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started
2023-03-29 17:27:53,375 INFO o.a.j.t.JMeterThread: Thread started: Thread Group 1-1
2023-03-29 17:27:53,379 INFO o.a.j.s.SampleResult: Note: Sample TimeStamps are START times
2023-03-29 17:27:53,380 INFO o.a.j.s.SampleResult: sampleresult.default.encoding is set to ISO-8859-1
2023-03-29 17:27:53,380 INFO o.a.j.s.SampleResult: sampleresult.useNanoTime=true
2023-03-29 17:27:53,380 INFO o.a.j.s.SampleResult: sampleresult.nanoThreadSleep=5000
2023-03-29 17:28:00,893 INFO o.a.j.t.JMeterThread: Thread finished: Thread Group 1-1
2023-03-29 17:28:00,893 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[Thread Group 1-1,6,main]
java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.poll(Ljava/time/Duration;)Lorg/apache/kafka/clients/consumer/ConsumerRecords;
    at com.di.jmeter.kafka.sampler.KafkaConsumerSampler.getConsumerRecords(KafkaConsumerSampler.java:90) ~[di-kafkameter-1.2.jar:?]
    at com.di.jmeter.kafka.sampler.KafkaConsumerSampler.sample(KafkaConsumerSampler.java:73) ~[di-kafkameter-1.2.jar:?]
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) ~[ApacheJMeter_core.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) ~[ApacheJMeter_core.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) ~[ApacheJMeter_core.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) ~[ApacheJMeter_core.jar:5.5]
    at java.lang.Thread.run(Thread.java:834) ~[?:?]
2023-03-29 17:28:00,916 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2023-03-29 17:28:00,923 INFO c.d.j.k.c.KafkaConsumerConfig: Kafka consumer client connection terminated
2023-03-29 17:28:00,924 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, *local*)

Here's the jmeter log: jmeter.log

About the topic name, yes i'm using di-kafkameter as topic name to test this plugin i already produced some message inside, so it's not an empty topic

rollno748 commented 1 year ago

Hey Thanks for the log. I could see that - the consumer config is properly initialised and its failing at the polling stage image

The error indicates that the JMeter is unable to find the method java.lang.NoSuchMethodError: org.apache.kafka.clients.consumer.KafkaConsumer.poll

So, the quick check you need to do is :

  1. Do you have the required dependent libraries present in <JmeterInstalledDirectory>/lib directory
  2. Check for the Duplicates/lower version of kafka-clients jar - If you find anything other than kafka-clients-3.3.1.jar , please remove it
  3. Uninstall the plugin and verify all the dependent libraries are removed and re-install it.
  4. If nothing of the above works - Try it on freshly installed JMeter

I will keep this open - until you confirm the above methods

hengkybuntoroalldata commented 1 year ago

Hooo, the error is fixed when i placed the depedencies to jmeter lib/ext folder

2023-03-29 18:21:03,229 INFO o.a.k.c.u.AppInfoParser: Kafka version: 3.1.1
2023-03-29 18:21:03,229 INFO o.a.k.c.u.AppInfoParser: Kafka commitId: 5cf0285dc9728b0b
2023-03-29 18:21:03,230 INFO o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1680088863229
2023-03-29 18:21:03,231 INFO o.a.k.c.c.KafkaConsumer: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Subscribed to topic(s): di-kafkameter
2023-03-29 18:21:03,231 INFO c.d.j.k.c.KafkaConsumerConfig: Kafka consumer client successfully Initialized
2023-03-29 18:21:03,233 INFO o.a.j.g.u.JMeterMenuBar: setRunning(true, *local*)
2023-03-29 18:21:03,322 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : Thread Group
2023-03-29 18:21:03,323 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group Thread Group.
2023-03-29 18:21:03,323 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error
2023-03-29 18:21:03,323 INFO o.a.j.t.ThreadGroup: Starting thread group... number=1 threads=1 ramp-up=1 delayedStart=false
2023-03-29 18:21:03,324 INFO o.a.j.t.ThreadGroup: Started thread group number 1
2023-03-29 18:21:03,325 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started
2023-03-29 18:21:03,326 INFO o.a.j.t.JMeterThread: Thread started: Thread Group 1-1
2023-03-29 18:21:03,364 INFO o.a.k.c.Metadata: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Resetting the last seen epoch of partition di-kafkameter-0 to 0 since the associated topicId changed from null to mMy1cILSST29q99bXaKgcg
2023-03-29 18:21:03,365 INFO o.a.k.c.Metadata: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Cluster ID: U35uatg0SwmQ3Gdnk-X_Gg
2023-03-29 18:21:03,365 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Discovered group coordinator docker-kafka-1:19092 (id: 2147483646 rack: null)
2023-03-29 18:21:03,368 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] (Re-)joining group
2023-03-29 18:21:03,397 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Request joining group due to: need to re-join with the given member-id
2023-03-29 18:21:03,398 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] (Re-)joining group
2023-03-29 18:21:06,410 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Successfully joined group with generation Generation{generationId=3, memberId='consumer-consumer-group-1-2-4405e72f-b246-4b5e-a9c4-b64c1107ddaf', protocol='range'}
2023-03-29 18:21:06,411 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Finished assignment for group at generation 3: {consumer-consumer-group-1-2-4405e72f-b246-4b5e-a9c4-b64c1107ddaf=Assignment(partitions=[di-kafkameter-0])}
2023-03-29 18:21:06,425 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Successfully synced group in generation Generation{generationId=3, memberId='consumer-consumer-group-1-2-4405e72f-b246-4b5e-a9c4-b64c1107ddaf', protocol='range'}
2023-03-29 18:21:06,425 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Notifying assignor about the new Assignment(partitions=[di-kafkameter-0])
2023-03-29 18:21:06,426 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Adding newly assigned partitions: di-kafkameter-0
2023-03-29 18:21:06,437 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Setting offset for partition di-kafkameter-0 to the committed offset FetchPosition{offset=10, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[docker-kafka-4:49092 (id: 4 rack: null)], epoch=0}}
2023-03-29 18:21:06,618 INFO o.a.j.t.JMeterThread: Thread is done: Thread Group 1-1
2023-03-29 18:21:06,618 INFO o.a.j.t.JMeterThread: Thread finished: Thread Group 1-1
2023-03-29 18:21:06,618 INFO o.a.j.e.StandardJMeterEngine: Notifying test listeners of end of test
2023-03-29 18:21:06,631 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Revoke previously assigned partitions di-kafkameter-0
2023-03-29 18:21:06,632 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Member consumer-consumer-group-1-2-4405e72f-b246-4b5e-a9c4-b64c1107ddaf sending LeaveGroup request to coordinator docker-kafka-1:19092 (id: 2147483646 rack: null) due to the consumer is being closed
2023-03-29 18:21:06,632 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Resetting generation due to: consumer pro-actively leaving the group
2023-03-29 18:21:06,633 INFO o.a.k.c.c.i.ConsumerCoordinator: [Consumer clientId=consumer-consumer-group-1-2, groupId=consumer-group-1] Request joining group due to: consumer pro-actively leaving the group
2023-03-29 18:21:06,647 INFO o.a.k.c.m.Metrics: Metrics scheduler closed
2023-03-29 18:21:06,647 INFO o.a.k.c.m.Metrics: Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-03-29 18:21:06,648 INFO o.a.k.c.m.Metrics: Metrics reporters closed
2023-03-29 18:21:06,656 INFO o.a.k.c.u.AppInfoParser: App info kafka.consumer for consumer-consumer-group-1-2 unregistered
2023-03-29 18:21:06,656 INFO c.d.j.k.c.KafkaConsumerConfig: Kafka consumer client connection terminated
2023-03-29 18:21:06,657 INFO o.a.j.g.u.JMeterMenuBar: setRunning(false, *local*)

so i need to place the dependencies library in lib/ext, my bad i thought the dependencies only needed for when i compile the code myself.

Thank you so much for the support, i think the issue can be closed

rollno748 commented 1 year ago

Glad, the error is fixed FYI: You can install the plugin via plugins manager as well. I have published it there as well for easy use.

Please hit a star to the repo, if you like the work. That means a lot !