confluent-tools.sh not found

[giampaolotrapasso]

I'm trying to run the kafka-avro-console-consumeras indicated in the Docker part of the guide. However I got this error:

giampaolo@rbbox:~/dev/bottledwater-pg$ !14
docker-compose run --rm kafka-avro-console-consumer --from-beginning --topic test
ERROR: Container command '/confluent-tools.sh' not found or does not exist.

I thought that the tools are installed with the kafka-toolsservice of the composed file, so I run `giampaolo@rbbox: ~/dev/bottledwater-pg$ docker-compose up kafka-tools`` that returns me

bottledwaterpg_zookeeper_1 is up-to-date
bottledwaterpg_kafka_1 is up-to-date
Creating bottledwaterpg_kafka-tools_1

ERROR: for kafka-tools  Container command '/confluent-tools.sh' not found or does not exist.
Attaching to 

The image sees correcty installed as I run:

giampaolo@rbbox:~/dev/bottledwater-pg$ docker ps -a
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS              PORTS                                        NAMES
b0191fba162a        confluent/tools                    "/confluent-tools.sh"    12 seconds ago      Created                                                          bottledwaterpg_kafka-tools_1
42b78d6ea2aa        bottledwaterpg_bottledwater-json   "/usr/local/bin/bottl"   About an hour ago   Up About an hour    5432/tcp                                     bottledwaterpg_bottledwater-json_1
52bebcdeb550        postgres:9.5                       "/docker-entrypoint.s"   About an hour ago   Up About an hour    5432/tcp                                     bottledwaterpg_psql_run_1
d3ec0e4eb95f        confluent/schema-registry          "/usr/local/bin/schem"   2 hours ago         Up 2 hours          0.0.0.0:48081->8081/tcp                      bottledwaterpg_schema-registry_1
cfd600441482        confluent/kafka:0.9.0.0-cp1        "/usr/local/bin/kafka"   2 hours ago         Up 2 hours          0.0.0.0:9092->9092/tcp                       bottledwaterpg_kafka_1
59cefed40549        confluent/zookeeper                "/usr/local/bin/zk-do"   2 hours ago         Up 2 hours          2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp   bottledwaterpg_zookeeper_1
2d4633b5d710        bottledwaterpg_postgres            "/docker-entrypoint.s"   2 hours ago         Up 2 hours          0.0.0.0:45432->5432/tcp                      bottledwaterpg_postgres_1

How can I fix this? thank you in advance

[samstokes]

Thanks for reporting this @giampaolotrapasso. It looks like Confluent published a new version of the confluent/tools Docker image that no longer has a single entrypoint like the previous version did, and thus no longer makes use of the environment variables set by container links (probably because those environment variables are deprecated). This breaks the documented command to run kafka-avro-console-consumer, and also more generally breaks our current Docker Compose setup.

It looks like you have a workaround for now? I'll try to work on fixing this.

The proper solution will be to get rid of all reliance on link environment variables and instead use the container hostnames, which should also work with future versions of Docker. That should be possible by putting the config right into docker-compose.yml.

[giampaolotrapasso]

Yes @samstokes, that commit resolve it as well. IIRC my fork resolves the 2 issues I posted and allowed me to make the playground working properly and to connect it to Flink. About this, I think that any client using Kafka outside the container will need the KAFKA_ADVERTISED_HOST_NAME parameter. I changed the compose file like this:

kafka:
  image: confluent/kafka:0.9.0.0-cp1
  hostname: kafka
  links:
    - zookeeper
  environment:
    KAFKA_LOG_CLEANUP_POLICY:
    KAFKA_ADVERTISED_HOST_NAME: $KAFKA_ADVERTISED
    KAFKA_AUTO_CREATE_TOPICS_ENABLE:
  ports:
    - '9092:9092'

and passed KAFKA_ADVERTISED variable within my environment.

[samstokes]

I've merged a fix on master.

@giampaolotrapasso re KAFKA_ADVERTISED_HOST_NAME: you are correct, that does need to be set in the host environment (see comment at the top of docker-compose.yml). However, you don't need to define a separate KAFKA_ADVERTISED variable name. From the Docker Compose documentation:

Environment variables with only a key are resolved to their values on the machine Compose is running on, which can be helpful for secret or host-specific values.

So you can do something like:

export KAFKA_ADVERTISED_HOST_NAME=172.17.0.1
docker-compose up -d kafka