search

apache / pekko-connectors-kafka

Apache Pekko Kafka Connector - Pekko-Connectors is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Apache Pekko.
https://pekko.apache.org/
Apache License 2.0
22 stars 12 forks source link

Apache Pekko based Streaming application is idle and not consuming from kafka topic post rebalancing #249

Open dahiyahimanshu opened 1 month ago

dahiyahimanshu commented 1 month ago

Issue: Apache Pekko-based Streaming Application Idle After Kafka Rebalancing

Description: Our application, which runs on Kubernetes with KEDA scaling based on consumer group lag, has encountered issues post-rebalancing. During testing, the application scales up and down as expected. However, after a rebalance, certain Kafka topic partitions are no longer being consumed.

We observed that while the consumer is assigned multiple partitions, some partitions are paused and never resumed, resulting in unprocessed messages. The application only resumes consuming these pending messages when there is another surge of input messages, which triggers pod scaling and, subsequently, another rebalance.

mdedetrich commented 1 month ago

Can you isolate whether this is a pekko-connectors-kafka issue or whether it's due to the underlying kafka client?

pjfanning commented 1 month ago

also, we need to know which version of pekko-connectors-kafka that you are using - it is recommended to use pekko-connectors-kafka 1.1.0 if you can

dahiyahimanshu commented 1 month ago

it is exactly the same issue as described here. https://github.com/akka/alpakka-kafka/pull/194

it is not underlying client issue but something to do with rebalance which pauses some partition but never resume them again, i checked KafkaConsumerActor file which has poll method where for some partitions we are not calling resume method. i checked this behaviour in debug logs.

i am using pekko-connectors-kafka 1.0.0 version. actually i am running with scala 2.12. Can i use 1.1.0 with scala 2.12?