avoid port binding issues running create topic shell script within the container

[ches]

Getting all the JMX settings is rather fiddly to get right. You're going to have to give an appropriately thorough description in your pull request about how this works, what problem this solves for you, etc.

[robwdux]

ches - I was looking for a working kafka container that could join an ensemble of zookeeper nodes and I didn't have any luck until trying yours. however attempting to verify beyond kafka registering with zk that it was indeed functional, I came up short attempting to create a topic when following the readme. I've corrected the issues encountered and tested them by creating an ensemble with 3 zk nodes and 3 kafka instances.

following the quick start, single kafka container talking with a zookeeper ensemble - I receive an exception attempting to create a topic within the instance to the zk ensemble (those doing the quickstart with a single node to try it out will likely encounter this issue).

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is: java.net.BindException: Address already in use

kafka-run-class.sh is aware of JMX_PORT the server script calls this script for which it then binds to 7203 as exported in the dockerfile to the environment. attempting to verify kafka is functional by running kafka-topics.sh to create a topic I receive the exception because this script calls kafka-run-class.sh which then attempts to spawn a process - instructed to bind to the same port declared by JMX_PORT and fails. Essentially, not exporting that variable solves the problem. My commits correct that problem as well as another. essentially, the KAFKA_JMX_OPTIONS don't get exported unless a value is passed during runtime given that it is being evaluated for length but isn't defined in this case.

[ches]

Thanks for the further explanation. I think I follow what you're getting at, but I'll need to make a few minutes to try to reproduce—can you give me an exact sequence of commands to reproduce your issue?

I believe my fundamental question is, how are you running two processes in a container such that they conflict on binding the port? I assume that's the only way this would happen, and it's not really intended usage, but I'll try to see if there's a reasonable way to accommodate.

following the quick start, single kafka container talking with a zookeeper ensemble - I receive an exception attempting to create a topic within the instance to the zk ensemble

Can you elaborate on what you mean by "within the instance"? I assume this has to do with how you're ending up with two processes competing for the port.

essentially, the KAFKA_JMX_OPTIONS don't get exported unless a value is passed during runtime given that it is being evaluated for length but isn't defined in this case.

I don't follow you. If the var is unset, then if [ -z "$KAFKA_JMX_OPTS" ]; then ... and if [[ -z ${KAFKA_JMX_OPTS:-} ]]; then ... are essentially equivalent:

$ unset BOZO_VAR
$ if [ -z "$BOZO_VAR" ]; then echo empty; fi
empty
$ if [[ -z ${BOZO_VAR:-} ]]; then echo empty; fi
empty

KAFKA_JMX_OPTS does get exported inside the conditional in start.sh as long as the user passes no explicit value for it at runtime, I've used this recently to connect VisualVM from outside containers.

[eliaslevy]

I believe my fundamental question is, how are you running two processes in a container such that they conflict on binding the port?

He is executing a single process in the container (Kafka), but is attempting to execute kafka-topic.sh within the container to verify the cluster state. That will result in the error he got if JMX_PORT is in the environment, as kafka-run-class.sh enables JMX support indiscriminetely and using the same JMX_PORT as the server within the server container will lead to a port conflict.

I run into the same issue on my own containers.

[greatpsi]

The changes outlined at the top of the post resolved the issue with the port binding. Further description of the build process follows:

Containers were built using docker-compose, where container_name elements were added for both zookeeper and kafka. Also made the following (partial) change to expose ZOOKEEPER_IP for the start.sh script:

kafka: build: . image: ches/kafka container_name: kafka links:

• zookeeper environment:
• ZOOKEEPER_IP=zookeeper

After docker-composer up -d and connecting the the kafka container using docker exec, the port binding problem occurred. Modifying the two files as indicated above allowed topics to be created, published and consumed from within the kafka container.

[markusthoemmes]

Running into the same problem. Very annoying. As a workaround:

docker exec kafka bash -c "unset JMX_PORT; your_scriptcall"

Is there still something blocking the merge of this? I'ld be happy to rebase the fix to master to make it mergeable again if you agree on the change @ches .

[bluepuma77]

kafka-topics.sh doesn't run out of the box.

# docker run \
  --name=kafka1 \
  -p 9092:9092 -p 7203:7203 \
  -e ZOOKEEPER_CONNECTION_STRING=server1.server.de:2181 \
  -e KAFKA_BROKER_ID=0 \
  -e KAFKA_DEFAULT_REPLICATION_FACTOR=2 \
  -e KAFKA_AUTO_CREATE_TOPICS_ENABLE=true \
  -e KAFKA_ADVERTISED_HOST_NAME=server1.server.de \
  -e KAFKA_ADVERTISED_PORT=9092 \
  -d ches/kafka

# docker exec -it kafka1 bash -c "/kafka/bin/kafka-topics.sh --zookeeper server1.server.de:2181 --list"
Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7203; nested exception is:
    java.net.BindException: Address already in use

The mentioned workaround unset JMX_PORT does work.

# docker exec -it kafka1 bash -c "unset JMX_PORT; /kafka/bin/kafka-topics.sh --zookeeper server1.server.de:2181 --list"
test

Shouldn't that be default? Or at least be mentioned in the docs?

[ches]

Ahhhhh so the problem here is docker exec, I wish that was made clear from the start. @robwdux asserted that he was following the README.

I came up short attempting to create a topic when following the readme… I receive an exception attempting to create a topic within the instance to the zk ensemble (those doing the quickstart with a single node to try it out will likely encounter this issue).

(Emphasis mine)—my README does not instruct people to use docker exec. Don't do that. TL;DR, use docker run.

He is executing a single process in the container (Kafka), but is attempting to execute kafka-topic.sh within the container to verify the cluster state.

Using docker exec means you are executing a second process in the container 😝

This is a feature of last resort for unusual debugging situations. 95% of the time people are using docker exec, they don't need to. You're effectively mutating the expected state of a running container instance, which I regard as antithetical to good Docker practices.

I will not accept the changes proposed here because:

1. The image was not designed with running multiple processes in the container in mind, and I don't intend to account for that as regular usage.
2. JMX_PORT is the environment variable expected by users of the canonical Kafka distribution, KAFKA_JMX_PORT is non-standard and not used by the official Kafka scripts. The JMX configuration is already confusing and I don't wish to further complicate understanding.
3. The image is behaving as expected. Download the Kafka tarball and follow the quick start, but export JMX_PORT=7203 in both the shell where you start the broker and the shell where you run kafka-topics.sh. You'll encounter the same issue. That's essentially what's happening with docker exec, you're getting the same shell environment in two processes that share a port.

There is a simple solution: use docker run --rm ches/kafka kafka-topics.sh, as documented.

If you really must exec for some reason, add --env JMX_PORT= (empty value).

By the way, you'll also likely encounter this problem if you run multiple container instances with --network=host. That's also a Docker feature of last resort and you're on your own to --env or remap ports.

[ches]

Something I'll consider that would make this less likely to be encountered: giving JMX_PORT an empty value by default so it's opt-in to enable, since that is the case with the Kafka distribution out of the box anyway.

Still though, break your docker exec habits 😅