search

wurstmeister / kafka-docker

Dockerfile for Apache Kafka
http://wurstmeister.github.io/kafka-docker/
Apache License 2.0
6.92k stars 2.73k forks source link

Topic creation overload #618

Open danielmotaleite opened 4 years ago

danielmotaleite commented 4 years ago

I'm doing a setup where i'm creating around 10 topics. When i start kafka, the machine load jumps to around 20, mostly because the topic creation script starts all the topic creation in parallel. So starting kafka and 10 commands is basically starting 11 java virtual machines at same time, a heavy operation for most machines. I would imagine some big live setup creating 50 topics killing almost any instance.

So probably it is better to either start those creation in serial mode instead of parallel or limit the number of parallel runs to maybe 3, to avoid overloading the machine on startup.

danielmotaleite commented 4 years ago

Probably related to https://github.com/wurstmeister/kafka-docker/issues/399

afternun commented 3 years ago

I am having the same issue, trying to add around 80 topics and that is killing or freezing my Kafka image. This is not a big problem, because I can use my custom script and run it synchronously inside of the container, but it adds additional step to the deployment. Limiting the number of parallel runs or option to run them synchronously would be great!

cbelogu commented 3 years ago

I am having the same issue with single broker setup. Trying to create roughly 12 topics and noNode issue happens. any suggestions pls ?