search

dpkp / kafka-python

Python client for Apache Kafka
http://kafka-python.readthedocs.io/
Apache License 2.0
5.52k stars 1.39k forks source link

Issues with Amazon MSK #2314

Open einarjohnson opened 2 years ago

einarjohnson commented 2 years ago

Hello, I am currently using version 2.0.2 to publish messages to an Amazon MSK broker version 2.3.1 but I am running into a wall when I get this exception (when calling the send method)

File \"/usr/local/lib/python3.8/site-packages/kafka/producer/kafka.py\", line 702, in _wait_on_metadata\n raise Errors.KafkaTimeoutError(\nkafka.errors.KafkaTimeoutError: KafkaTimeoutError: Failed to update metadata after 60.0 secs." My code works perfectly when I run it locally using the confluentinc/cp-kafka:5.4.3 docker image as a Kafka broker and I am wondering if there are some compatibility issues with this MSK version mentioned above. I notice that 2.3 isn't in the compatibility matrix. I have verified that the topic, which I am publishing to, exists on the broker as this was the first thing suggested when I googled this issue. Also verified that the container running the producer can connect to the MSK cluster (by telnetting to the 9092 port)

einarjohnson commented 2 years ago

ignore me, figured out what the problem was. the container was reading the topic name from an env var which was mangled and I totally oversaw this.