spotify / docker-kafka

Kafka (and Zookeeper) in Docker
Apache License 2.0
1.4k stars 642 forks source link

Note that server.properties lost few comment lines in 0.10.0 #40

Open gwenshap opened 8 years ago

gwenshap commented 8 years ago

Please update your start script to avoid failures due to these lines: https://github.com/spotify/docker-kafka/blob/master/kafka/scripts/start-kafka.sh#L20

advertised.host and #advertised.port were removed from Kafka's default file.

We have #listeners now.

jshark commented 8 years ago

Hi,

I made the following changes in order to support the 0.10.0 version, with advertised_listeners instead of advertised.host/port.

Probably will be useful for someone..

diff --git a/kafka/scripts/start-kafka.sh b/kafka/scripts/start-kafka.sh
index ff37918..1a0b344 100755
--- a/kafka/scripts/start-kafka.sh
+++ b/kafka/scripts/start-kafka.sh
@@ -17,12 +17,17 @@ fi
 # Set the external host and port
 if [ ! -z "$ADVERTISED_HOST" ]; then
     echo "advertised host: $ADVERTISED_HOST"
-    sed -r -i "s/#(advertised.host.name)=(.*)/\1=$ADVERTISED_HOST/g" $KAFKA_HOME/config/server.properties
-fi
-if [ ! -z "$ADVERTISED_PORT" ]; then
-    echo "advertised port: $ADVERTISED_PORT"
-    sed -r -i "s/#(advertised.port)=(.*)/\1=$ADVERTISED_PORT/g" $KAFKA_HOME/config/server.properties
+    #sed -r -i "s/#(advertised.host.name)=(.*)/\1=$ADVERTISED_HOST/g" $KAFKA_HOME/config/server.properties
+    if [ -z "$ADVERTISED_PORT" ]; then
+      sed -r -i "s/#(advertised.listeners)=(PLAINTEXT:\/\/)([^:]*)(:[0-9]*)/\1=\2${ADVERTISED_HOST}\4/g" $KAFKA_HOME/config/server.properties
+    else
+      sed -r -i "s/#(advertised.listeners)=(PLAINTEXT:\/\/)([^:]*)(:[0-9]*)/\1=\2${ADVERTISED_HOST}:${ADVERTISED_PORT}/g" $KAFKA_HOME/config/server.properties
+    fi
 fi
+# if [ ! -z "$ADVERTISED_PORT" ]; then
+#     echo "advertised port: $ADVERTISED_PORT"
+#     sed -r -i "s/#(advertised.port)=(.*)/\1=$ADVERTISED_PORT/g" $KAFKA_HOME/config/server.properties
+# fi

 # Set the zookeeper chroot
 if [ ! -z "$ZK_CHROOT" ]; then
JonathanAaron commented 8 years ago

@jshark I tried adding this to my start-kafka.sh. The sed command works as expected, but the only way I can call kafka from outside of my container is to add advertised.host.name and advertised.port to server.properties. 0.10.0.0 doesn't have it so the original sed command doesn't work either. Is your machine working with your script?

jshark commented 8 years ago

@JonathanAaron if I understand you correctly, you don't specify the environment variable ADVERTISED_HOST/PORT when you start your container. And you should. Here is an example from my docker-compose file:

version: '2'
  services:
     kafka:
       image: 'my_services/kafka:latest'
       hostname: kafka.server
       environment:
         ADVERTISED_HOST: kafka.server
         ADVERTISED_PORT: 9092
       ports:
       - 2181:2181
       - 9092:9092

You can also use the CLI option to specify it.

GalaxyGorilla commented 7 years ago

@jshark, your comment from above saved my day! I have a slightly complicated setup with a dedicated docker host and it absolutely did not work without proper use of the hostname option when I updated from 1.5.x to 3.2.0. I think your example compose file earns a place in the README, I just found this place by coincidence.