search

sclasen / akka-kafka

185 stars 62 forks source link

User's persistent failure to acknowledge messages results in endless "state=Committing msg=StateTimeout drained=0 streams=1" #2

Closed acruise closed 10 years ago

acruise commented 10 years ago

Hi there,

I've been using akka-kafka 0.0.4 in a little prototype project, and have found that after an initial burst of messages being processed successfully, I get a seemingly endless succession of these lines in my log, about every 1020 milliseconds. It's entirely possible I'm just doing it wrong, but my use case is really straightforward, so I doubt it. :)

acruise commented 10 years ago

On closer inspection, it looks like I wasn't replying to the sender with StreamFSM.Processed. I'll leave the issue open but change the title.

sclasen commented 10 years ago

@acruise thanks for the feedback! This is pretty much by design. You basically need to handle every message and respond with StreamFSM.Processed or akka-kafka will not commit the offsets since it doesn't know if you have processed a given message.

At commit time, the Connector waits for all the streams to finish processing any in-flight messages. Once that happens successfully, the offsets are committed, and you can be sure you have processed any message up to the committed offset for a given partition.

Not to say that your business logic has to succeed for every message, but you need to have error handling baked into that logic, that for instance sends the message to an errors topic in kafka.