Updated td-agent 3.2.0 is silently failing

[jurgenhaas]

Since the update to td-agent 3.2.0 for which I wasn't able to find any changelog yet, nothing is being processed anymore and not even the logs contain any details. Has anyone else experienced that as well and could point me in the right direction?

The plugins being used and successfully loaded during startup are:

    - 'fluent-plugin-secure-forward'
    - 'fluent-plugin-multi-format-parser'
    - 'fluent-plugin-record-modifier'
    - 'fluent-plugin-elasticsearch'
    - 'fluent-plugin-elasticsearch-timestamp-check'
    - 'fluent-plugin-kvp-filter'

[fujimotos]

I have updated my testing td-agent instance to v3.2.0 and can confirm that it processes events without any problem (Note: I have tested this instance with tail/http/forward inputs and all worked fine).

Evidently, your issue is something specific to your environment. So, to investigate the issue further, could you do:

1. Post your full configuration file here, and
2. Enable debug logging on your instance and paste the resutling log.

You can follow this procedure to enable verbose logging.

[repeatedly]

Since the update to td-agent 3.2.0

Does this mean update from 3.1.1 to 3.2.0, right? You may check full logs via https://docs.fluentd.org/v1.0/articles/trouble-shooting#check-uncaught-logs

[jurgenhaas]

I've given this another try and still don't get any output from td-agent 3.2.0.

@repeatedly yes, I am updating from 3.1.1 and I don't have any uncaught logs.

@fujimotos I have created this Gist which is a short log with trace being enabled. It contains the full configuration right at the top.

This doesn't create any buffered files nor does it forward anything to the host 9.8.7.6 where the very same configuration works perfectly fine with version 3.1.1 of td-agent.

What else can I try to find out what's going wrong?

[fujimotos]

@jurgenhaas Your log suggests that out_forward does not receive any record at all. Otherwise you should see a line "enqueueing all chunks in buffer" on shutdown.

The strong suspect is grep filter. I'm guessing that the pluin does not work as you expect and filters all events out.

Can you run your instance without the following block and report back me what happens?

<filter **>
  @type grep
  <exclude>
    key "agent"
    pattern Python-urllib|Uptime|Varnish Health Probe
  </exclude>
  <exclude>
    key "path"
    pattern /server-status\?auto
  </exclude>
</filter>

[jurgenhaas]

@fujimotos you're right, that's the issue. However, it used to work as expected in 3.1.1 - so do you have any idea what has changed in 3.2.0 and how I could still filter out those documents?

[fujimotos]

Replace the configuration block with the following one and see what happens:

<filter **>
  @type grep
  <exclude>
    key "agent"
    pattern Python-urllib|Uptime|Varnish Health Probe
  </exclude>
  <exclude>
    key "path"
    pattern \/server-status\?auto
  </exclude>
</filter>

Evidently, your issue is caused by the changes introduced in Fluentd v1.2.0.

Since the version, the second exclude block (somehow) gets translated to RegExp(//), which effectively matches everything. This is why no event reaches to the subsequent match block.

The above configuration should work as you expect.

[jurgenhaas]

That seems to be fixing it. Thanks a lot @fujimotos for your help on this.