Add support for avoid throttle by fields

[shaikatz]

Hi, First of all, thanks for your great plugin, I find it very useful and powerful.

Our use case requires group records by pod_name + specific log_levels. For example, we want to throttle only records that have log_level warn or error.

Thought that it'll be nice to have an option to specify field and value to prevent the record from being throttled. it can be built like record_modifier does it:

<filter pattern>
  @type record_modifier

  # replace key key1
  <replace>
    # your key name
    key key1
    # your regexp
    expression /^(?<start>.+).{2}(?<end>.+)$/
    # replace string
    replace \\k<start>ors\\k<end>
  </replace>
  # replace key key2
  <replace>
    # your key name
    key key2
    # your regexp
    expression /^(.{1}).{2}(.{1})$/
    # replace string
    replace \\1ors\\2
  </replace>
</filter>

For this plugin we can have:

<ignore_record>
   key log_level
   expression /(info|debug|trace)/
</ignore_record>

What do you think?

[shaikatz]

Since I didn't get any comment about this issue, I've implemented the feature. We currently consider forking the plugin and publishing it as a separate ruby gem since we're in need of this functionality here at my company. Naturally - it will be better to merge this change into your plugin to keep everything in one place, please let me know if this is something you're in favor of too.

[g3kr]

@shaikatz would be able to share what you have done?

[shaikatz]

@g3kr you can find it in my fork https://github.com/shaikatz/fluent-plugin-throttle sorry for the late response.