search

cloudstax / firecamp

Serverless Platform for the stateful services
https://www.cloudstax.io
Apache License 2.0
208 stars 21 forks source link

Kafka configuration changes #77

Open rik2803 opened 5 years ago

rik2803 commented 5 years ago

Hi,

Is there a way to customize the Kafka configuration? I tried logging into the all (3) containers and updating /data/conf/server.properties but that file is reset to its original content after the container is restarted when I run the restart-service command. (and I do not understand why/how, but that's not the question).

And the FireCamp documentation does not seem to mention how to change the configuration.

Kind regards, Rik

JuniusLuo commented 5 years ago

The current behavior is by design. FireCamp tries to expose the minimal number of configs. We would like to collect the requirements first before adding more configs or adding the general mechanism to allow updating custom configs. What kind of configurations do you plan to change?

The config files inside container is immutable. FireCamp loads the configs that are input at the service creation time, and overwrites the /data/conf/server.properties file before container starts. When container starts, entrypoint.sh will load the configs in /data/conf/server.properties and update kafka configs accordingly.

JuniusLuo commented 5 years ago

A simple common mechanism is to allow tuning the non-sensitive configs via the environment variables. This is not complex to support. If you could provide more details, we could consider to support it.

timchurchard commented 5 years ago

I would like to set auto.create.topics.enable=false would you consider supporting this?

JuniusLuo commented 5 years ago

This is a good config to support. Will add it.

nagaraj07 commented 5 years ago

With respect to above, configuration changes. If I want edit, server.properties of kafka to change below listeners to any other IP/DNS:

listeners=PLAINTEXT://mykafka-0.abc-firecamp.com:9092 advertised.listeners=PLAINTEXT://mykafka-0.abc-firecamp.com:9092

Once, we change this if i restart the kafka and zookeeper service it again goes to default config. How to override this config? Or how to pass this parameters as environment variable to override the values while creating new service?

JuniusLuo commented 5 years ago

Changing the listeners address is not supported. The kafka container binds with the DNS name. This is how failover works. When the kafka container moves to another node, DNS will be automatically updated.