fluent-plugins-nursery / fluent-plugin-systemd

This is a fluentd input plugin. It reads logs from the systemd journal.
Apache License 2.0
153 stars 43 forks source link

Segmentation fault reading from systemd #71

Closed kentsteer closed 5 years ago

kentsteer commented 5 years ago

I'm seeing continuous segmentation faults on STDERR, seemingly when trying to initialize the journal:

/var/lib/gems/2.3.0/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:230: [BUG] Segmentation fault at 0x00000000050b20
ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]

-- Control frame information -----------------------------------------------
c:0013 p:---- s:0056 e:000055 CFUNC  :sd_journal_open_directory
c:0012 p:0102 s:0050 e:000049 METHOD /var/lib/gems/2.3.0/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:230
c:0011 p:0046 s:0042 e:000041 METHOD /var/lib/gems/2.3.0/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:51 [FINISH]
c:0010 p:---- s:0034 e:000033 CFUNC  :new
c:0009 p:0027 s:0030 e:000029 METHOD /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:68
c:0008 p:0009 s:0026 e:000025 METHOD /var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:107 [FINISH]
c:0007 p:---- s:0023 e:000022 IFUNC 
c:0006 p:0014 s:0021 e:000020 METHOD /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/timer.rb:80 [FINISH]
c:0005 p:---- s:0017 e:000016 CFUNC  :run_once
c:0004 p:0042 s:0013 e:000012 METHOD /var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88
c:0003 p:0033 s:0009 e:000008 BLOCK  /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/event_loop.rb:93
c:0002 p:0081 s:0006 e:000005 BLOCK  /var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/thread.rb:78 [FINISH]
c:0001 p:---- s:0002 e:000001 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
/var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
/var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run'
/var/lib/gems/2.3.0/gems/cool.io-1.5.3/lib/cool.io/loop.rb:88:in `run_once'
/var/lib/gems/2.3.0/gems/fluentd-1.2.5/lib/fluent/plugin_helper/timer.rb:80:in `on_timer'
/var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:107:in `run'
/var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:68:in `init_journal'
/var/lib/gems/2.3.0/gems/fluent-plugin-systemd-1.0.1/lib/fluent/plugin/in_systemd.rb:68:in `new'
/var/lib/gems/2.3.0/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:51:in `initialize'
/var/lib/gems/2.3.0/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:230:in `open_journal'
/var/lib/gems/2.3.0/gems/systemd-journal-1.3.3/lib/systemd/journal.rb:230:in `sd_journal_open_directory'

While STDOUT is showing the following repeatedly (with pid changing):

2018-11-01 00:18:00 +0000 [info]: fluent/log.rb:322:info: Worker 0 finished unexpectedly with signal SIGSEGV
2018-11-01 00:18:00 +0000 [info]: fluent/log.rb:322:info: gem 'fluent-plugin-systemd' version '1.0.1'
2018-11-01 00:18:00 +0000 [info]: fluent/log.rb:322:info: gem 'fluentd' version '1.2.5'
2018-11-01 00:18:00 +0000 [info]: fluent/log.rb:322:info: adding source type="systemd"
2018-11-01 00:18:00 +0000 [info]: #0 fluent/log.rb:322:info: starting fluentd worker pid=1304 ppid=25 worker=0

Source config:

<source>
  @type systemd
  filters [{ "_SYSTEMD_UNIT": "docker.service" }]
  <storage>
      @type local
      persistent true
      path /var/log/docker.log.pos
  </storage>
  tag docker
</source>

Let me know if there are any other details I can provide or experiments I can run.

errm commented 5 years ago

Are you running fluentd on the same host as the journal is being created on or in a docker container?

If you are in a container could you share the version of systemd on the host and in your container?

errm commented 5 years ago

Closing since there isn't much we can do without knowing specific versions!