What is the best way to check if Connection is there and if not retry it

[Nabil-Miri]

Description

Hello, I have a code with a producer. I want to know if there is a method to know if there is any error or no connection in the producer and if not, there reconnect or create the producer again.

Code:

How to reproduce

  def checkAndReconnect(self):
      while True:  # stay in the loop until it is reconnected!!
          try:
              if not self.kafkaProducer.poll(0):
                  # Try to reconnect
                  # Create a new producer
                  self.kafkaProducer = Producer(self.config)  # creating a new producer to connection again
                  break  # Exit the loop if reconnection is successful
              else:
                  break
          except KafkaException as ke:
              kafka_error = ke.args[0]
                print(
                    "Error: KafkaException during reconnection: " + str(kafka_error),
                    flush=True,
                )

Currently the poll(0) does not help in doing so. What could be a solution? Thanks.

Checklist

Please provide the following information:

• [ ] confluent-kafka-python and librdkafka version (confluent_kafka.version() and confluent_kafka.libversion()): 2.5.0
• [ ] Apache Kafka broker version:
• [ ] Client configuration: {...}
• [ ] Operating system: Ubuntu 22.04
• [ ] Provide client logs (with 'debug': '..' as necessary)
• [ ] Provide broker log excerpts
• [ ] Critical issue

[pranavrth]

Producers internally tries to reconnect periodically if the connection is not established. You don't need to do that on your own.

[Nabil-Miri]

So no need for the whole checkAndReconnect() function, right?

[Nabil-Miri]

@pranavrth BTW, I noticed in some places, reconnection is tested using list_topics like in the test_Producer.py

    try:
        p.list_topics(timeout=0.2)
    except KafkaException as e:
        assert e.args[0].code() in (KafkaError._TIMED_OUT, KafkaError._TRANSPORT)

So what is the difference if I used the list_topics and the periodic reconnection of the Producer.

[pranavrth]

Producer internally tries to connect to the brokers periodically. You will achieve the same thing if you try create the producer again and again with the same bootstrap-servers. Creating producer instance is a costly operation and will be an overhead for your application.

[Nabil-Miri]

Noted. Thanks a lot for your reply.