So I currently I have a usecase where I need to set up a kafla cluster with multiple brokers (i.e. at least 2) to diagnose a performance issue with reassigning partitions. Since I am setting up multiple brokers I need to use the HOSTNAME_COMMAND to manually set the hostname for each broker.
This is the initial docker-compose.yaml that I used
Now I need to actually set up a JMX port so that I can connect to the the broker instances using visualvm so I adjusted the docker-compose.yaml to look like the following
The main issue here is using the _{HOSTNAME_COMMAND} in -Djava.rmi.server.hostname for the KAFKA_JMX_OPTS. With these changes and using docker-compose up -d --scale kafka=2 there seems to be some interpolation error when kafka tries to start, I get the following error
/usr/bin/start-kafka.sh: line 59: export: `-Dcom.sun.management.jmxremote.authenticate=false': not a valid identifier
/usr/bin/start-kafka.sh: line 59: export: `-Dcom.sun.management.jmxremote.ssl=false': not a valid identifier
/usr/bin/start-kafka.sh: line 59: export: `-Djava.rmi.server.hostname=192.168.160.1': not a valid identifier
/usr/bin/start-kafka.sh: line 59: export: `-Dcom.sun.management.jmxremote.rmi.port=1099': not a valid identifier
so the actual substitution for _{HOSTNAME_COMMAND} seems to work since 192.168.160.1 is in the error but it appears that the substitution method is somehow breaking the -D java settings that get sent to kafka. Note that if I manually put in 192.168.160.1 into the docker-compose, i.e.
The cluster works fine but I then have issues to the host using visualvm, i.e. after a run of docker-compose up -d --scale kafka=2 and then doing docker ps -a I get
a5c10f9bd8bb wurstmeister/kafka:latest "start-kafka.sh" 25 seconds ago Up 24 seconds 0.0.0.0:49222->1099/tcp, 0.0.0.0:49221->9092/tcp docker-compose_kafka_1
beec603b54cc wurstmeister/kafka:latest "start-kafka.sh" 25 seconds ago Up 24 seconds 0.0.0.0:49220->1099/tcp, 0.0.0.0:49219->9092/tcp docker-compose_kafka_2
d414d7e5d557 wurstmeister/zookeeper:latest "/bin/sh -c '/usr/sb…" 26 seconds ago Up 25 seconds 22/tcp, 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp docker-compose_zookeeper_1
The first brokers JXM host should be localhost:49220 however in visualvm I don't seem to be able to connect to this
So I currently I have a usecase where I need to set up a kafla cluster with multiple brokers (i.e. at least 2) to diagnose a performance issue with reassigning partitions. Since I am setting up multiple brokers I need to use the
HOSTNAME_COMMAND
to manually set the hostname for each broker.This is the initial
docker-compose.yaml
that I usedNow I need to actually set up a JMX port so that I can connect to the the broker instances using visualvm so I adjusted the
docker-compose.yaml
to look like the followingThe main issue here is using the
_{HOSTNAME_COMMAND}
in-Djava.rmi.server.hostname
for theKAFKA_JMX_OPTS
. With these changes and usingdocker-compose up -d --scale kafka=2
there seems to be some interpolation error when kafka tries to start, I get the following errorso the actual substitution for
_{HOSTNAME_COMMAND}
seems to work since192.168.160.1
is in the error but it appears that the substitution method is somehow breaking the-D
java settings that get sent to kafka. Note that if I manually put in192.168.160.1
into the docker-compose, i.e.The cluster works fine but I then have issues to the host using visualvm, i.e. after a run of
docker-compose up -d --scale kafka=2
and then doingdocker ps -a
I getThe first brokers JXM host should be
localhost:49220
however in visualvm I don't seem to be able to connect to this