bencoughlan
commented
7 years ago KAFKA_AUTO_CREATE_TOPICS_ENABLE=true needs to be set.
However, I've found that this opens up a whole new world of pain, as other apps will attach with the broker's default settings, it's a security hole too.
What I Confluent recommend is to
A - bring cluster up with this turned on. B - Once topic's are auto created. turn it off, and recycle all the brokers C - Lift the firewall and allow apps to connect.
It's a terribly unpractical set of steps to make, particularly on a large cluster.
Start up the cluster with this enabled in a dev environment.
Describe all the topic's. Script the topic creation with the kafka-topic's tool. Then start up the environment without this setting enabled, and run the script after the environment(all zk's and brokers) comes up.
Good luck!
The process of creating property files in confluent images is bit opaque. Looks like there are some custom binaries involved (dub and cub) to interpolate the property file templates with environment variables.
Can the image be made flexible to enable/set additional Kafka server properties ? Would be great if someone could help setting the auto.create.topics.enable to true in the current images.