Bug: kafka testcontainer does not support adding a network

[champialex]

Describe the bug

KafkaContainer.tc_start only allows containers to have a single IP, so it is not possible to add a network

Changing listeners = f"PLAINTEXT://{host}:{port},BROKER://$(hostname -i):9092" to listeners = f"PLAINTEXT://{host}:{port},BROKER://$(hostname -i | cut -d' ' -f1):9092" fixes it

Exception in the container logs:

java.lang.IllegalArgumentException: Error creating broker listeners from 'PLAINTEXT://192.168.64.2:33647,BROKER://172.17.0.3 172.21.0.2:9092': Unable to parse BROKER://172.17.0.3 172.21.0.2:9092 to a broker endpoint
    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:266)
    at kafka.server.KafkaConfig.effectiveAdvertisedListeners(KafkaConfig.scala:2154)
    at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2275)
    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2233)
    at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1604)
    at kafka.Kafka$.buildServer(Kafka.scala:72)
    at kafka.Kafka$.main(Kafka.scala:91)
    at kafka.Kafka.main(Kafka.scala)
Caused by: org.apache.kafka.common.KafkaException: Unable to parse BROKER://172.17.0.3 172.21.0.2:9092 to a broker endpoint
    at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:58)
    at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$10(CoreUtils.scala:263)
    at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
    at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
    at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37)
    at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:263)
    ... 7 more

To Reproduce

    from kafka import KafkaAdminClient
    from testcontainers.kafka import KafkaContainer

    with Network() as network:
        kafka_ctr = KafkaContainer()
        kafka_ctr.with_network(network)
        kafka_ctr.with_network_aliases("kafka")

        with kafka_ctr:
            print("started") # Will not reach here and timeout
            admin_client = KafkaAdminClient(bootstrap_servers=[kafka_ctr.get_bootstrap_server()])
            print(admin_client.describe_cluster())

Runtime environment

[[package]] name = "testcontainers" version = "4.7.1"

# Get the operating system information (on a unix os).
$ uname -a
Darwin sydmacg366 23.3.0 Darwin Kernel Version 23.3.0: Wed Dec 20 21:31:10 PST 2023; root:xnu-10002.81.5~7/RELEASE_ARM64_T6031 arm64
# Get the python version.
$ python --version
Python 3.12.2
# Get the docker version and other docker information.
$ docker info

Client: Docker Engine - Community
 Version:    27.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     /Users/adecham/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  2.28.1
    Path:     /Users/adecham/.docker/cli-plugins/docker-compose

Server:
 Containers: 338
  Running: 1
  Paused: 0
  Stopped: 337
 Images: 24
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 0cae528dd6cb557f7201036e9f43420650207b58
 runc version: 860f061b76bb4fc671f0f9e900f7d80ff93d4eb7
 init version:
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 6.1.55-0-virt
 Operating System: Alpine Linux v3.18
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 7.753GiB
 Name: colima
 ID: c74f716c-594e-4e7c-8ae1-f9f71efbd2ac
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[alexanderankin]

seems related to #633

can you share more of the use case, i understand this is a complete reproduction but i am also kind of curious the use case, whether in python form or a general description - I do not use this feature myself so I want to understand a bit better. thanks.

[champialex]

Thanks for the quick turnaround!

The use case is when chaining several containers (in my case telegraf and kafka). Having a network alias makes it possible to do that without having to do complicated file generation based on actual IPs: I can just point telegraf to kafka:9092.