search

fluent-plugins-nursery / fluent-plugin-systemd

This is a fluentd input plugin. It reads logs from the systemd journal.
Apache License 2.0
153 stars 43 forks source link

systemd plugin fails to recognise journal file rotation #27

Closed evmin closed 7 years ago

evmin commented 7 years ago

RHEL 7.3 systemd plugin 0.1.0

It looks like the plugin stops feeding the entries on journal file rotation.

How to reproduce:

update /etc/systemd/journald.conf with the config entries (just for demo sake only):

[Journal] Storage=persistent SystemMaxUse=8M SystemMaxFileSize=1M SystemMaxFiles=3 MaxRetentionSec=1d MaxFileSec=5m

reload systemd config: systemctl force-reload systemd-journald

Once the file is rotated after 5 min the logs stop flowing.

evmin commented 7 years ago

Mine is on td-agent 2.3.4-0.el7 - fluentd 0.14.11 if I am not mistaken.

And one more thing worth noting. Since I have disabled the log file rotation - i.e. system.journal would start anew when filled, the behaviour have been inconsistent - sometimes the plugin would happily continue capturing the entries, but sometimes would halt.

Was not able to identify the pattern as yet.

errm commented 7 years ago

Hi @evmin I am trying to take a look at this issue this week, are you seeing any errors in the output from fluentd, or is it just a case of this failing silently?

taskinen commented 7 years ago

I am too seeing this issue. There are no errors in td-agent.log. Only this gets printed while the streaming from journald is frozen:

2017-02-08 06:25:01 +0200 [info]: flushing all buffer forcedly 2017-02-08 06:25:01.000000000 +0200 fluent.info: {"message":"force flushing buffered events"} 2017-02-08 06:25:01.000000000 +0200 fluent.info: {"message":"flushing all buffer forcedly"}

The definitive hint about the reason of this freeze is that it happens exactly when journald log is rotated.

errm commented 7 years ago

Is this something that showed up in version 0.1.0 or is it an issue with the 0.0.x branch as well?

errm commented 7 years ago

@evmin, @taskinen I have released v0.1.1 and v0.0.7 that should resolve this issue... could you see if it does indeed resolve your issue before I close this ticket.