Process finished code=134

[andrericos]

hi,

I tried to use this plugin to read my app's log, but i had some issues.

First of all, there is a Dockerfile that a created to show my current progress:

FROM centos:7

RUN rpm --import https://packages.treasuredata.com/GPG-KEY-td-agent \
      && printf "[treasuredata]\nname=TreasureData\nbaseurl=http://packages.treasuredata.com/2/redhat/\$releasever/\$basearch\ngpgcheck=1\ngpgkey=https://packages.treasuredata.com/GPG-KEY-td-agent\n" > /etc/yum.repos.d/td.repo \
      && yum install -y td-agent make gcc-c++ systemd

ENV PATH /opt/td-agent/embedded/bin/:$PATH

RUN fluent-gem install fluent-plugin-systemd -v 0.0.4

#until now, nothing new. It's just a copy from test Dockerfile

#appending a systemd source with tag kube-proxy (it's is similar to documentation sample)
RUN printf "<source>\ntype systemd\npath /run/log/journal\nfilters [{ \"_SYSTEMD_UNIT\": \"td-agent.service\" }]\npos_file kube-proxy.pos\ntag kube-proxy\nread_from_head true\n</source>" >> /etc/td-agent/td-agent.conf

#Without that line the td-agent does not read any data from journey. It seems to be havving some prigilegy issue
RUN usermod -G systemd-journal td-agent
#With that line the td-agent seems to access the journal data, but dies after that.

CMD /usr/sbin/init

#docker run --privileged -t fluenttest:1 .
#docker exec -it  .... bash
#$ systemctl restart td-agent

Initially, i tried to follow the documentation and just add this in /etc/td-agent/td-agent.conf:

<source>
  type systemd
  path /var/log/journal
  filters [{ "_SYSTEMD_UNIT": "kube-proxy.service" }]
  pos_file kube-proxy.pos
  tag kube-proxy
  read_from_head true
</source>

It didn't work and i realized that the journalctl was putting the data in /run/log/journal, because some persistence property.

then a update the code to:

<source>
  type systemd
  path /run/log/journal
  filters [{ "_SYSTEMD_UNIT": "kube-proxy.service" }]
  pos_file kube-proxy.pos
  tag kube-proxy
  read_from_head true
</source>

it didn't work yet!

Then, i saw the td-agent service is executed by a user called td-agent and i tried test if this user has privileges to access those files. I run:

root@localhost$ sudo -u td-agent ls -la /run/log/journal/fe65ef0463ab46989303f72fcbc58aef
ls: cannot open directory '/run/log/journal/fe65ef0463ab46989303f72fcbc58aef': Permission denied

Then i included td-agent into systemd-journal group (owner of /run/log/journal).

After that, the td-agent service seems to be reading the journal, but dies after the first read.

There is the log:

2016-09-21 16:59:42 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-09-21 16:59:42 +0000 [info]: starting fluentd-0.12.26
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.13'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-s3' version '0.6.8'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-systemd' version '0.0.4'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-09-21 16:59:42 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-09-21 16:59:42 +0000 [info]: gem 'fluentd' version '0.12.26'
2016-09-21 16:59:42 +0000 [info]: adding match pattern="td.*.*" type="tdlog"
2016-09-21 16:59:42 +0000 [info]: adding match pattern="debug.**" type="stdout"
2016-09-21 16:59:42 +0000 [info]: adding source type="forward"
2016-09-21 16:59:42 +0000 [info]: adding source type="http"
2016-09-21 16:59:42 +0000 [info]: adding source type="debug_agent"
2016-09-21 16:59:42 +0000 [info]: adding source type="systemd"
2016-09-21 16:59:42 +0000 [info]: using configuration file: <ROOT>
  <match td.*.*>
    @type tdlog
    apikey xxxxxx
    auto_create_table 
    buffer_type file
    buffer_path /var/log/td-agent/buffer/td
    <secondary>
      @type file
      path /var/log/td-agent/failed_records
      buffer_path /var/log/td-agent/failed_records.*
    </secondary>
  </match>
  <match debug.**>
    @type stdout
  </match>
  <source>
    @type forward
  </source>
  <source>
    @type http
    port 8888
  </source>
  <source>
    @type debug_agent
    bind 127.0.0.1
    port 24230
  </source>
  <source>
    type systemd
    path /run/log/journal
    filters [{"_SYSTEMD_UNIT":"td-agent.service"}]
    pos_file kube-proxy.pos
    tag kube-proxy
    read_from_head true
  </source>
</ROOT>
2016-09-21 16:59:42 +0000 [info]: listening fluent socket on 0.0.0.0:24224
2016-09-21 16:59:42 +0000 [info]: listening dRuby uri="druby://127.0.0.1:24230" object="Engine"
2016-09-21 16:59:42 +0000 [warn]: no patterns matched tag="kube-proxy"  -----> It did read something from journal!!!! <-----
2016-09-21 16:59:42 +0000 [info]: process finished code=134
2016-09-21 16:59:42 +0000 [warn]: process died within 1 second. exit.

I have no idea what is wrong...

[errm]

It seems this is a dupe of #8 it is caused by a dependency that needed updating in td-agent.

As a workaround you could try installing ruby and the fluentd gem yourself. Here is my current production setup https://github.com/assemblyline/fluentd/blob/master/templates/Dockerfile.erb

[andrericos]

It works!

thanks!