search

SahilKang / cl-rdkafka

Common Lisp library for Kafka
Other
42 stars 7 forks source link

Something in the Kafka chain is not raising conditions #57

Closed kat-co closed 4 years ago

kat-co commented 4 years ago

Hello, and thank you for the project! Our Kafka environment is as follows: v0.10.0.0 Broker -> v1.3.0 librdkafka -> cl-rdkafka-20200218-git (quicklisp).

We have a long-lived process which continually polls/commits. Occasionally, we see this message logged:

%4|1583796667.962|MAXPOLL|rdkafka#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 226ms (adjust max.poll.interval.ms for long-running message processing): leaving group

(As an aside, adjusting the max poll interval does not cause this not to happen. As far as we can tell, the default is right-sized for our workload.)

Shouldn't this kind of thing be raised as a condition in cl-rdkafka?

SahilKang commented 4 years ago

cl-rdkafka will signal a condition if librdkafka signals an error in polling; that rd-kafka-message->message function is called by the consumer poll method.

I found this comment by librdkafka's original author indicating that it's just a log message instead of an error condition. As to why you might be seeing it, do you think your consumer is waiting for a long time on an empty topic by any chance?

kat-co commented 4 years ago

Interesting; I wonder at why it's a log message instead of an error. Isn't the result that a call to poll will block indefinitely? I wonder if cl-rdkafka should raise a condition or return a sentinel value when poll is invoked but the client has left the consumer group?

We found out that we were exiting the consumer group because of #58. It is not as logically correct, but we have enabled auto.commit which sidesteps the problem.

Thanks for the input! I'll close this although I'm happy to continue the conversation :)