search

confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
110 stars 895 forks source link

AdminClient.create_topics call hanging #833

Open jpelletier-dad opened 4 years ago

jpelletier-dad commented 4 years ago

Description

Calls to AdminClient.create_topic() seem to spin indefinitely (even with request_timeouts added). This is after previous calls to retrieve the topics succeed (so the configuration would appear to be correct).

How to reproduce

CODE:

    def create_topics(self, topics, partitions):
        new_topics = [
            confluent_kafka.admin.NewTopic(t.topic, partitions, 2) for t in topics
        ]
        # futures == dict(<topic_name, future>)
        futures = confluent_kafka.admin.AdminClient(self.conf).create_topics(
            new_topics, request_timeout=5
        )

        for topic, f in futures.items():
            try:
                # blocking
                f.result()
                print("Topic {} created".format(topic))
            except Exception as e:
                print("Failed to create topic {}: {}".format(topic, e))
        return

Checklist

Please provide the following information:

jpelletier-dad commented 4 years ago

Still experiencing. I updated librdkafka-dev and am using the test release 

>>> confluent_kafka.version()
('1.4.0.1', 17039616)
>>> confluent_kafka.libversion()
('1.4.0', 17039615)

I'm running this in python:3.8-alpine Docker container with these dependencies

apk --no-cache add \
  gcc \
  musl-dev \
  librdkafka-dev@edgecommunity~=1.4.0

From my limited ability to grok the logs i get the impression that the broker connection is being killed before the create topics requests can go through.

jfinzel commented 4 years ago

We are having the same issue. But strange thing is it works for me, but not for another co-worker. Please advise!!!