search

dpkp / kafka-python

Python client for Apache Kafka
http://kafka-python.readthedocs.io/
Apache License 2.0
5.58k stars 1.4k forks source link

Why does producer asks for describing every topic in cluster? #2244

Open Pyrrha opened 3 years ago

Pyrrha commented 3 years ago

Hello,

I received a lot of logs, where I see that kafka-python try to describe every topic of the cluster, with Denied result. I don't know where this list come from, and why the package use it and ask for a describe.

Do you know where I can disable the "auto-discovery" feature of the package? It's on kafka-python==1.4.6. The documentation doesn't seem to indicate something about it.

Thanks!

lquastana commented 3 years ago

Hi ,

We have the same issue ... @Pyrrha , have you found a solution for this problem ?

Thank you

lquastana commented 3 years ago

We find a solution , if you set bootstrap_topics_filter parameter with the topics needed, the producer will only call the describes on the specified topics. For the consumer its a little bit tricky because the argument is not accepted by the constructor ....

Pyrrha commented 3 years ago

Hello @lquastana, this subject wasn't a priority so we didn't dig a lot since my post. Moreover, I went in vacation... Thanks for your advice, I'll add this modification when back at the office, and look after a solution for the consumer part.

dinegri commented 2 years ago

It is happening in our cluster as well. We have almost 2000 topics, because of that we noticed a latency increase in kafka request handler because it is writing in the log4j files authorizer 1999 denied describe access(we changed for ERROR)…We’ll try to use bootstrap_topics_filter