search

sclasen / akka-kafka

185 stars 62 forks source link

ConsumerTimeoutException #43

Closed mjuchli closed 8 years ago

mjuchli commented 8 years ago

I see millions of Exceptions being catched here: https://github.com/sclasen/akka-kafka/blob/457a9ec6323b23c15affeb3cb182e811f574c830/src/main/scala/com/sclasen/akka/kafka/Actors.scala#L184

We're using the default consumer.timeout of 400ms at this point. Do I understand correctly, that if there is no message within a period of 400ms, the iterator will throw this exception? What is the best practice in order to prevent exceptions?, just increase the timeout?

Thanks! Marc

sclasen commented 8 years ago

@backender those exceptions are an expected part of normal operations. are they causing you any issue?

if you increase the timeout, your commit latencies will increase as well, since committing offsets has to wait for those iterator polls to time out, if the stream is not actively receiving messages.

mjuchli commented 8 years ago

It does not particularly causes a problem–it's just that we inject exception listeners in our JVM's and the ConsumerTimeoutException being raised go into the millions. So obviously I was wondering what's the reason for this.

Why was this chosen over let's say having a mutable state?

sclasen commented 8 years ago

hi @backender this is just the way the stock high-level consumer that ships with kafka works. akka-kafka just wraps this consumer and orchestrates it so that there are no unprocessed messages in-flight before committing.

did you have an idea for how to do this without polling with a timeout?

if you have a topic where every partition sees regular traffic, and have the number of streams = the number of partitions, i would not expect to see that many ConsumerTimeoutExceptions.

How many streams are you using across the consumer group in comparison to the number of partitions in your topic?

mjuchli commented 8 years ago

hi @sclasen, I missed that this is kafka high-level consumer behaviour and not especially related to akka-kafka. From what I've read (http://mail-archives.apache.org/mod_mbox/kafka-users/201408.mbox/%3CCACim9RkJ6+6jJKzP3UQKQPbsOUsboifBoySpOYPm=2RPp9XUcw@mail.gmail.com%3E) this is a downside this blocking iterator brings with. This "flaw" is supposed to be fixed in 0.9.

You were right, I indeed had more streams than partitions on the topic. I'm curious if the number of exceptions will decrease now.

I think we can close the issue. At least we have extended the documentation around this here :)

sclasen commented 8 years ago

:+1: