search

RackSec / desdemona

Data-backed security operations
Eclipse Public License 1.0
2 stars 7 forks source link

Kafka container prematurely exits about half of the time #45

Closed ehashman closed 8 years ago

ehashman commented 8 years ago

We saw an example of this in our latest Travis build on my Docker PR (here), and I've also encountered this in my dev environment. We should track down what the problem is and fix it if possible.

Current workaround is to run docker-compose up again until the Kafka container doesn't die.

ehashman commented 8 years ago

Logs say:

[2016-02-18 18:56:18,519] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path /brokers/ids/1. This probably indicates that you either have configured a brokerid that is already in use, or else you have shutdown this broker and restarted it faster than the zookeeper timeout so it appears to be re-registering.

so @sirsean's inclination that this was a Zookeeper issue sounds about right to me.

lvh commented 8 years ago

I seem to remember someone mentioning that they had found more details on this somewhere. If that's the case, please update this ticket.

ehashman commented 8 years ago

I suspect that this bug has something to do with Kafka's interactions with Zookeeper. From the kafka-users mailing list: http://mail-archives.apache.org/mod_mbox/kafka-users/201305.mbox/%3CFF142F6B499AE34CAED4D263F6CA32901CAA9F88@EXTXMB19.nam.nsroot.net%3E

I also see an instance of this error related to non-graceful shutdown that a PR exists for upstream on the container image that we're using, but that doesn't seem to describe the clean creation case: https://github.com/wurstmeister/kafka-docker/pull/58

ehashman commented 8 years ago

Closed by #85.