search

lensesio / fast-data-dev

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, , 20+ connectors
https://lenses.io
Apache License 2.0
2.02k stars 331 forks source link

How to pass custom config options to Kafka #39

Open teckays opened 7 years ago

teckays commented 7 years ago

For example I would like my topics to be created automatically, therefore, I'd like to somehow configure auto.create.topics.enable=true. Based on the documentation, it doesn't seem like there's a way to pass in this option as an environment variable, therefore I have tried the following:

-v ./docker/landoop/opt/confluent/etc/kafka:/opt/confluent/etc/kafka

Then add the auto.create.topics.enable=true in the server.properties file.

However, the above causes some weird behaviour on the Kafka Broker, here are the logs:

landoop_1       | 2017-09-28 11:53:03,886 INFO exited: broker (exit status 1; not expected)
landoop_1       | 2017-09-28 11:53:04,749 INFO spawned: 'broker' with pid 281
landoop_1       | 2017-09-28 11:53:05,851 INFO success: broker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
landoop_1       | 2017-09-28 11:53:11,867 INFO exited: rest-proxy (exit status 1; not expected)
landoop_1       | 2017-09-28 11:53:12,604 INFO spawned: 'rest-proxy' with pid 314
landoop_1       | 2017-09-28 11:53:13,024 INFO exited: schema-registry (exit status 1; not expected)
landoop_1       | 2017-09-28 11:53:13,735 INFO spawned: 'schema-registry' with pid 341
landoop_1       | 2017-09-28 11:53:13,739 INFO success: rest-proxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

which obviously repeats over time when trying to restart it.

Plus, when I go to http://localhost:3030/kafka-topics-ui/, I see the following:

KAFKA REST
/api/kafka-rest-proxy 
CONNECTIVITY ERROR

Thank you.

chadgrant commented 6 years ago

would like this as well ...

andmarios commented 6 years ago

Hi. This is actually possible now, please have a look here: https://github.com/Landoop/fast-data-dev#configure-kafka-components

As an example, to set auto.create.topics.enable=true (though this is the default setting anyway), you would add the env var:

KAFKA_AUTO_CREATE_TOPICS_ENABLE=true

Keep in mind this works only with the latest versions of fast-data-dev (tags latest, 1.0, 1.0.1, 1.1, 1.1.0). The latest tag is the same as 1.0 and 1.0.1, the current stable version.