search

fluent-plugins-nursery / fluent-plugin-systemd

This is a fluentd input plugin. It reads logs from the systemd journal.
Apache License 2.0
153 stars 43 forks source link

Error: Is a directory @ io_getpartial - /var/log/journal #103

Open a-x- opened 1 year ago

a-x- commented 1 year ago

newrelic recommends fluentd as a log forwarder. So, i'm trying to setup it... I just need to pass all the systems logs I get this ^ error in newrelic

/etc/td-agent/td-agent.conf ``` # installation: td-agent-gem install fluent-plugin-systemd -v 1.0.3 # Caddy from journald @type systemd tag caddy path /var/log/journal matches [{ "_SYSTEMD_UNIT": "caddy.service" }] read_from_head true @type local path /var/log/fluentd/journald-caddy-cursor.json fields_strip_underscores true fields_lowercase true # Newrelic Fluentd log plugin # installation: td-agent-gem install fluent-plugin-newrelic # Forward to Newrelic @type newrelic license_key XXX base_uri https://log-api.eu.newrelic.com/log/v1 @type memory flush_interval 5s root_dir /var/log/fluentd ```
/var/log/td-agent/td-agent.log ``` 2023-04-05 12:35:53 +0000 [info]: starting fluentd-1.15.3 pid=6838 ruby="3.1.2" 2023-04-05 12:35:53 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"] 2023-04-05 12:35:53 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil 2023-04-05 12:35:53 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil 2023-04-05 12:35:53 +0000 [info]: adding match pattern="**" type="newrelic" 2023-04-05 12:35:53 +0000 [warn]: #0 'flush_interval' is ignored because default 'flush_mode' is not 'interval': 'lazy' 2023-04-05 12:35:53 +0000 [info]: adding match pattern="debug.**" type="stdout" 2023-04-05 12:35:53 +0000 [info]: adding source type="systemd" 2023-04-05 12:35:54 +0000 [warn]: #0 detect empty plugin storage file during startup. Ignored: /var/log/fluentd/journald-caddy-cursor.json 2023-04-05 12:35:54 +0000 [info]: adding source type="forward" 2023-04-05 12:35:54 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead. 2023-04-05 12:35:54 +0000 [info]: adding source type="unix" 2023-04-05 12:35:54 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead. 2023-04-05 12:35:54 +0000 [info]: adding source type="http" 2023-04-05 12:35:54 +0000 [info]: adding source type="debug_agent" 2023-04-05 12:35:54 +0000 [warn]: #0 define to capture fluentd logs in top level is deprecated. Use

My setup:

# install logger dependency: https://docs.fluentd.org/installation/install-by-deb
# increase file descriptors
echo "# https://docs.fluentd.org/installation/before-install"  >> /etc/security/limits.conf
echo "root soft nofile 65536" >> /etc/security/limits.conf
echo "root hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
reboot
# td-agent 4 (experimental)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh | sh
systemctl status td-agent.service
a-x- commented 1 year ago

also, I don't understand what the *cursor.json and Is it plugin work w/o kubernetes (I saw kubelet in readme)

errm commented 1 year ago

Hi @a-x- thanks for the issue.

The storage records the offset of the journal cursor - this means if fluentd shuts down for some reason - then this plugin should be able to keep reading logs from where it left off.

This plugin doesn't directly intergrate with kuberntes - but of course you could run a container with a fluentd setup with this plugin in kubernetes - and as the readme shows you could tail the logs of some kuberentes components (if you were running them with systemd)