Closed dhawal55 closed 7 years ago
errm
commented
7 years ago Some things I would check first...
1) Do you have journal files in /var/log/journal ?
2) Is /var/log/journal mounted into whatever container you are using correctly
3) Does the user you are running fluentd as have the correct permission to read /var/log/journal
4) Is there a mismatch between the version of libsystemd writing the journal on the host, and inside your fluentd container ...
harobed
commented
7 years ago My docker-compose:
fluentd:
image: harobed/fluentd-with-plugins:latest
command: fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins --log /fluentd/log/fluentd.log
restart: unless-stopped
ports:
- 5140:5140/udp
- 9880:9880
volumes:
- ./fluentd-config/:/fluentd/etc/
- ./fluentd-log/:/fluentd/log/
- ./fluentd-pos/:/fluentd/pos/
- /var/log/journal/:/var/log/journal/
networks:
- elkfluentd configuration:
<system>
log_level debug
</system>
<source>
@type systemd
path /var/log/journal
<storage>
@type local
persistent true
path /fluentd/log/journald.pos
</storage>
tag journal
read_from_head false
</source>
<match **>
@type copy
<store>
@type file
path /fluentd/log/output
buffer_type file
buffer_path /fluentd/log/output.buffer
append false
flush_interval 5s
</store>
<store>
@type elasticsearch
host elasticsearch
buffer_type memory
flush_interval 1s
logstash_dateformat %Y%m%d
port 9200
logstash_format true
type_name fluentd
index_name logstash
include_tag_key true
tag_key _key
buffer_chunk_limit 512k
reload_connections false
reconnect_on_error true
max_retry_wait 60
disable_retry_limit
</store>
</match>my fluentd dockerfile:
FROM fluent/fluentd:v0.14-debian-onbuild
RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev" \
&& apt-get update \
&& apt-get install -y --no-install-recommends $buildDeps \
&& sudo gem install \
fluent-plugin-elasticsearch \
fluent-plugin-systemd \
&& sudo gem sources --clear-all \
&& SUDO_FORCE_REMOVE=yes \
apt-get purge -y --auto-remove \
-o APT::AutoRemove::RecommendsImportant=false \
$buildDeps \
&& rm -rf /var/lib/apt/lists/* \
/home/fluent/.gem/ruby/2.3.0/cache/*.gem/etc/systemd/journald.conf config
SystemMaxUse=400M
SystemMaxFileSize=100M
Storage=persistent
SyncIntervalSec=30sOS:
# cat /etc/debian_version
8.8fluentd have root access in Docker container.
# docker-compose exec fluentd ls /var/log/journal/8e26deeb0bd34f98a71d544e8005065b -lha
total 8.1M
drwxr-sr-x 2 root 102 4.0K Jul 18 11:52 .
drwxr-sr-x 3 root 102 4.0K Jul 18 11:52 ..
-rw-r----- 1 root 102 8.0M Jul 18 13:02 system.journalAfter:
# curl http://127.0.0.1:5601/ -I
HTTP/1.1 200 OK
kbn-name: kibana
kbn-version: 5.5.0
cache-control: no-cache
Date: Tue, 18 Jul 2017 13:06:53 GMT
Connection: keep-alivewith journalctl -xf I see:
Jul 18 15:06:53 server-elk dockerd[25573]: {"type":"response","@timestamp":"2017-07-18T13:06:53Z","tags":[],"pid":8,"method":"head","statusCode":200,"req":{"url":"/","method":"head","headers":{"user-agent":"curl/7.38.0","host":"127.0.0.1:5601","accept":"*/*"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"HEAD / 200 2ms - 9.0B"}nothing in fluentd-log/output.20170718_6.log and nothing in ElasticSearch.
Same subject here: https://groups.google.com/forum/#!topic/fluentd/CDsC-3ildpQ and https://fluent-all.slack.com/archives/C0CTT63EE/p1500377777737071
Best regards, Stéphane
harobed
commented
7 years ago Fixed with
environment:
- FLUENT_UID=0in my docker-compose.yml
errm
commented
7 years ago @dhawal55 did you manage to fix your issue?
cihangirbesiktas
commented
7 years ago I could not get the logs with the following configuration in Centos 7 host.
<system>
log_level debug
</system>
<source>
@type systemd
tag journal
path /var/log/journal
read_from_head true
<storage>
@type local
persistent true
path /var/log/td-agent/journal.pos
</storage>
</source>
<match journal>
@type file
path /tmp/fluentd/journal.json
format json
include_time_key true
time_key @timestamp
timezone +00:00
include_tag_key true
tag_key log_source
</match>
MattMencel
commented
7 years ago @cihangirbesiktas did you figure this out? I'm having the same issue on centos 7.
cihangirbesiktas
commented
7 years ago @MattMencel yes, it is due to td-agent user not having enough permissions, after I changed the service user as root, it worked properly.
MattMencel
commented
7 years ago Ah good catch. Might be better to add td-agent to the systemd-journal group than to run it as root? https://serverfault.com/questions/717725/journalctl-access-for-non-root-users
I'm unable to determine my fluentd is not reading my journal logs:
I'm using v0.2.0 of fluent-plugin-systemd and here's my fluentd config:
When I look at fluentd logs, everything looks fine but no journal logs are read:
I'm running this on CoreOS v1437.3.0