Option to re-emit unmatched records

[gshively11]

Related to https://github.com/fluent/fluent-plugin-rewrite-tag-filter/issues/73

Not emitting records when their tag doesn't change, even when they've matched a rule, was unexpected. I was trying to use a catchall rule to move logs to the next label/phase of the pipeline (using @label in the match):

  <rule>
      key _catch_all_
      pattern ^.+$
      invert true
      tag sometag
    </rule>

where sometag was the record's original tag. it would be nice if there was a configureable option to either cause a matched-but-unchanged tag to be re-emitted, or better yet, just have an option to remit all records regardless of patten match.

[okkez]

Unlimited re-emitting will cause an infinite loop easily. If this plugin re-emit events which is matched but not rewriten its tag, it will cause infinite loop. So we must be careful writing <rule> sections.

The following example may help you to construct routing.

<source>
  @type dummy
  dummy [
    {"message": "This is message", "source": "source1"},
    {"message": "This is message", "source": "source2"},
    {"message": "This is message", "source": "source3"},
    {"message": "This is message", "source": "source-none"}
  ]
  tag dummy
  @label @INPUT
</source>

<label @INPUT>
  <match dummy>
    @type rewrite_tag_filter
    <rule>
      key source
      pattern /^(source\d+)$/
      tag ${tag}.$1
    </rule>
    <rule>
      key source
      pattern /^.+$/
      tag unmatched
    </rule>
    @label @NEXT
  </match>
</label>

<label @NEXT>
  <match dummy.*>
    @type relabel
    @label @MATCHED
  </match>

  <match unmatched>
    @type relabel
    @label @UNMATCHED
  </match>
</label>

<label @MATCHED>
  <filter>
    @type record_transformer
    <record>
      matched true
    </record>
  </filter>
  <match>
    @type stdout
  </match>
</label>

<label @UNMATCHED>
  <filter>
    @type record_transformer
    <record>
      matched false
    </record>
  </filter>
  <match>
    @type stdout
  </match>
</label>

[gshively11]

Yep, we do something similar already. I was just hoping there could be an optional config that would allow match-but-unchanged, or unmatched, to be re-emitted. Using labels, as in your example, would prevent the infinite loop. The workaround we're using now involved changing the tag each phase, even though we'd prefer it to stay the same.

[okkez]

We cannot add the feature that re-emit both unmatched events and unmodified events, even if it is optional. Because it will cause an infinite loop easily, and it is a dangerous feature for all users.

[gshively11]

Ok, thanks for your time. I think you're overestimating the danger of an optional config, with sufficient documentation, that allows matched-but-unmodified events to be re-emitted. It would take a very specific configuration to get to that point in the first place.