Description:
Getting the following exception when using kafka:Consumer to consume messages from a Kafka topic concurrently.
The error:
Exception in thread "balx-kafka-consumer-network-thread" java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access. currentThread(name: balx-kafka-consumer-network-thread, id: 58) otherThread(id: 54)
at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:2551)
at org.apache.kafka.clients.consumer.KafkaConsumer.acquireAndEnsureOpen(KafkaConsumer.java:2532)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1234)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1227)
at io.ballerina.stdlib.kafka.nativeimpl.consumer.Poll.lambda$pollPayload$1(Poll.java:99)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
This is because even thought the kafka:Consumer is isolated, the underlying java consumer is not threadsafe. Therefore, it has to be synchronized manually.
Description: Getting the following exception when using
kafka:Consumer
to consume messages from a Kafka topic concurrently.The error:
Steps to reproduce:
Start a Kafka broker instance
Run the Ballerina service
public type Movie record {| string title; int year; string director; |};
service / on new http:Listener(8080) { private final kafka:Producer producer; private final kafka:Consumer consumer;
}