search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
72 stars 3.5k forks source link

Configuring log4j2.properties? #11622

Open sandeep1-0 opened 4 years ago

sandeep1-0 commented 4 years ago

I have a case of using logstash and kubernetes. So the issue i'm facing is with the logging part. I have mounted the folder where logs are being created to a host folder. So if run two instances of logstash both containers are creating logs to the same file mounted to the host folder. So is there any way to solve this like making changes in log4j2.properties so that dynamically it will create files based on container id or something unique factors. Did anyone came across a requirement like this?

andsel commented 4 years ago

You have the setting path.logs to setup different output directories (https://www.elastic.co/guide/en/logstash/current/dir-layout.html). You can specify it in logstash.yml or as CLI switch --path.logs

koalalam commented 4 years ago

You have the setting path.logs to setup different output directories (https://www.elastic.co/guide/en/logstash/current/dir-layout.html). You can specify it in logstash.yml or as CLI switch --path.logs

@andsel , correct. But when we want to add/disable certain appender, we have to provide a different configuration file. Would be good if there exists a config param where we can provide the location of the log4j2 config file e.g. --log.config=/path/to/log4j2.properties

Also, would be nice if the default log4j2.properties (with TimeBasedTriggeringPolicy and SizeBasedTriggeringPolicy enabled in rolling file appender) ships with a strategy that will clean up the old log files

andsel commented 4 years ago

From https://github.com/elastic/logstash/blob/7c6fca84256ee6d5dc98f34ace838a9d7d364e74/logstash-core/lib/logstash/runner.rb#L284-L285 they could set 2 different path.settings one for each LS instance, in both folder symlinking the common part and keep different log4j2.properties

koalalam commented 4 years ago

@andsel - good to know the standard log4j system property is supported as well. Would be helpful if this is mentioned in the doc: https://www.elastic.co/guide/en/logstash/current/logging.html

koalalam commented 4 years ago

@andsel - tried the following but got this issue: https://github.com/elastic/logstash/issues/8744

LS_JAVA_OPTS=-Dlog4j.configurationFile=/tmp/logstash_config/log4j2.properties /tmp/dist/logstash-6.3.2/bin/logstash --path.config /tmp/logstash_config/logstash.conf --path.data /tmp/logstash_data --path.logs /tmp/logstash_logs

2020-08-04 18:51:12,780 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root" 2020-08-04 18:51:12,783 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root" 2020-08-04 18:51:12,783 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog" 2020-08-04 18:51:12,784 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"