Kafka: Consumer interrupted with WakeupException after timeout.

[lmlynik]

I'm trying to run the example locally but I'm receiving lots of warns

[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds

This usually indicates kafka client to server mismatch, but can't find the exact version of kafka used here.

I'm using 0.10.1.0-1 from https://hub.docker.com/r/wurstmeister/kafka/tags/ but can switch quickly to different version, but can't pinpoint the version used by this example.

Thank you for your help

[ignasi35]

Hi @lmlynik that warn appears when Kafka server can't be reached. There's two scenarios there: Kafka takes time to start (the message you posted appears few times and then disappears) or some issue prevents Kafka from starting at all (the message you pasted continues appearing until an error is displayed in the logs).

The former can be addressed with a more powerful machine but, while it's inconvenient, it is not really an issue. The later is something we would like to investigate further. In the meantime we've posted a workaround and added a trace when the embedded Kafka fails to start: https://github.com/lagom/lagom/issues/382

All the info you can provide wrt your platform will be helpful when trying to get this issue solved or minimised.

[TimMoore]

@lmlynik Lagom's development environment runs its own Kafka + ZooKeeper services by default, you don't need to run it separately in Docker or otherwise. The internal ones are probably hitting port conflicts on startup so they aren't being used on your machine.

Could you let us know if you still have problems if you stop your Docker container(s)?

[neverfox]

I have this problem without Docker. All I did was clone this repo (a few minute ago), did a runAll and watched everything compile. At that point things were fine. I then stop the services (by hitting enter) and then did runAll again to have it start clean now that it was pre-compiled. At that point, I got spammed by the Kafka warnings:

> $ sbt                                                                                              ⬡ 7.5.0 [±master ✓]
[info] Loading project definition from /Users/neverfox/Repos/Clones/online-auction-scala/project
[info] Set current project to online-auction-scala (in build file:/Users/neverfox/Repos/Clones/online-auction-scala/)
> runAll
[info] Starting Kafka
[info] Starting Cassandra
........
[info] Cassandra server running at 127.0.0.1:4000
Kafka Server closed unexpectedly.
[info] Service locator is running at http://localhost:8000
[info] Service gateway is running at http://localhost:9000
[info] Service searchImpl listening for HTTP on 0:0:0:0:0:0:0:0:55320
[info] Service webGateway listening for HTTP on 0:0:0:0:0:0:0:0:60496
[info] Service userImpl listening for HTTP on 0:0:0:0:0:0:0:0:56181
[info] Service biddingImpl listening for HTTP on 0:0:0:0:0:0:0:0:58427
[info] Service itemImpl listening for HTTP on 0:0:0:0:0:0:0:0:53203
[info] (Services started, press enter to stop and go back to the console...)
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
^C[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
^C^C^C[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds
[warn] a.k.KafkaConsumerActor - Consumer interrupted with WakeupException after timeout. Message: null. Current value of akka.kafka.consumer.wakeup-timeout is 3000 milliseconds

Looks like the Kafka server crashes almost immediately.

[neverfox]

I did a clean, compile then runAll, and everything worked fine, even after stopping and doing a second runAll. Still probably worth nothing if this is something that tends to happen to people the first time out.

[TimMoore]

If anyone else runs into this problem, it would be helpful if you can provide the Kafka log.

You can find it at target/lagom-dynamic-projects/lagom-internal-meta-project-kafka/target/log4j_output/server.log upload it to a gist and paste a link to the gist into a comment here. Thanks!

[marykomar]

I have the same problem, running online-auction-java. Here is the link to gist: https://gist.github.com/marykomar/b88c1f01c4a2646be7f711b65f1414c7

[ignasi35]

Thanks @marykomar.

@neverfox, @marykomar, @lmlynik the Kafka server used in Development mode is actually a separate process started from sbt that will run zookeeper and kafka. You can see on your local machine using jps or ps aux that there's one process running the KafkaServer exclusively. The reason why somethimes the embedded Kafka server will fail to start are related to corrupted data/ files in your target/ folder. Those files are created and used by the embedded Kafka/Zookeeper combo internally but stored in the projects target/ that's why sbt clean usually will fix the problem.

From @marykomar's gist:

java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/0. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.

it looks like Zookeeper didn't restart cleanly.

We'll need to see why and try to avoid it.

Thanks for all the info.

[szymon-baranczyk]

I've run into the same problem while using some Activator template - now all my Lagom projects fail to launch (including online-auction-scala). I've tried sbt clean, restarting my machine and creating new project, nothing helped.

I've posted more info here: http://stackoverflow.com/questions/42675442/lagoms-embedded-kafka-fails-to-start-after-killing-lagom-process-once. You can see Kafka logs from two launches here: https://gist.github.com/szymonbaranczyk/a93273537b42aafa45bf67446dd41adb

Is it possible that some corrupted Embedded Kafka's data is stored globally on my pc and causes this?

[ignasi35]

This is related to http://stackoverflow.com/questions/42675442/lagoms-embedded-kafka-fails-to-start-after-killing-lagom-process-once

[AndyMoreland]

Is there an easy way to clean just this folder with sbt? This happens to me every time I restart sbt runAll and it's super annoying.

[omidb]

Any update on this issue? It seems if the target Kafka folder gets deleted, it will work fine but it doesn't happen automatically.