edenhill / kcat

Generic command line non-JVM Apache Kafka producer and consumer
Other
5.43k stars 484 forks source link

Problems consuming with kafkacat running in docker on WSL2 #301

Open mikaello opened 3 years ago

mikaello commented 3 years ago

I got some troubles consuming with kafkacat running in docker on WSL2.

Software versions:

Example:

# Start Kafka and create a topic, e.g. :
$ confluent local services kafka start
$ kafka-topics --create --topic test --bootstrap-server localhost:9092

$ PRIVATE_IP=$(ip addr show eth0 | grep 'inet\b' | awk '{print $2}' | cut -d/ -f1)

# Metadata with kafkacat local (does also work with broker localhost:9092), SUCCESS:
$ /usr/bin/kafkacat -L -b $PRIVATE_IP:9092 -t test
Metadata for test (from broker -1: 172.25.113.126::9092/bootstrap):
 1 brokers:
  broker 0 at my-computer-mikaello:9092 (controller)
 1 topics:
  topic "test" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0

# Metadata with kafkacat docker, SUCCESS:
$ docker run -it --network=host edenhill/kafkacat:1.6.0 -L -b $PRIVATE_IP:9092 -t test
Metadata for test (from broker -1: 172.25.113.126:9092/bootstrap):
 1 brokers:
  broker 0 at my-computer-mikaello:9092 (controller)
 1 topics:
  topic "test" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0

# Consume with kafkacat local, SUCCESS:
$ /usr/bin/kafkacat -C -b $PRIVATE_IP:9092 -t test -e
% Reached end of topic test [0] at offset 0: exiting

# Consume with kafkacat docker, FAILURE:
$ docker run -it --network=host edenhill/kafkacat:1.6.0 -C -b $PRIVATE_IP:9092 -t test -e
%3|1614806333.215|FAIL|rdkafka#consumer-1| [thrd:my-computer-mikaello:9092/0]: my-computer-mikaello:9092/0: Failed to resolve 'my-computer-mikaello:9092': Name does not resolve (after 3ms in state CONNECT)
% ERROR: Topic test [0] error: Failed to query logical offset END: Local: Host resolution failure

Is there any obvious reasons why this fails?

Since I'm trying to consume Avro, I need the docker version of kafkacat. Producing with kafkacat running in docker does not work either.

whatsupbros commented 3 years ago

In fact, you don't need docker version to consume Avro, but you need to build kafkacat with libavro and libserdes support. The bootstrap.sh script does this all for you, but you will have to track that all dependencies were built successfully.

Bu trial and error method, I found out that you'll need some additional packages for that, listed in #303