wurstmeister / kafka-docker

Dockerfile for Apache Kafka
http://wurstmeister.github.io/kafka-docker/
Apache License 2.0
6.92k stars 2.73k forks source link

how to change kafka log path #729

Open luoluo2001108 opened 1 year ago

luoluo2001108 commented 1 year ago

I change the logs.dir in server.properties ,but does not take effect.the value will be change after each reboot with docker-compose down/up,,why ?

log.dirs=/kafka/kafka-logs-56a3b229e3a1---change the log.dirs does not take effect

how to fix the log path ,even reboot kafka by docker-compose down/up ?

pseacrest commented 1 year ago

Same question herre

uriinf commented 1 year ago

You can use KAFKA_LOG_DIRS environment variable to override detault

opencmit2 commented 1 year ago

The default storage path for kafka configuration files is tmp/kafka logs. If you modify the kafka configuration file, the data storage path for kafka is as follows: log.dirs=/mnt/kafka/data

In addition to storing data, there are also Kafka's running log files, so we'd better also migrate them to a large disk. The kafka log file is stored in the logs folder of the kafka installation directory by default. In v2.7.1 and earlier versions, modifications cannot be directly defined in the configuration file, but only the running file/kafka/bin/kafka run class.sh can be modified

Since log.dirs is in the plural form, it actually supports configuring multiple paths, because Kafka can "load evenly" store data in multiple directories. Of course, it must be mounted to multiple disks instead of multiple directories on the same disk. Attaching multiple disks and allowing multiple heads to perform write operations simultaneously will greatly improve the throughput of Kafka. Kafka load balancing is measured relative to the number of partitions in Kafka, rather than based on disk space.

For example, you can specify two different disks (mnt1, mnt2) as the data storage directory, and modify the configuration file as follows:

log.dirs=/mnt1/kafka/data,/mnt2/kafka/data