search

tagomoris / fluent-plugin-parser

Other
74 stars 23 forks source link

Allow to preserve only part of the original record #39

Closed martin21 closed 7 years ago

martin21 commented 7 years ago

As described in issue https://github.com/fluent/fluent-bit/issues/261 I tried to use Fluent Bit to transfer Apache logs. I used ParserFilter plugin to parse Apache logs, as it seems that Fluent Bit is not yet able to do it.

Before that I let Fluentd add the hostname of the logsource into the logsource field (using this name to avoid a name clash with "host" field that Format apache seems uses for client host name). The ParserFilter removes that field again then.

I know I could use preserve_data yes, but it would keep the complete old record including the complete log message which Fluent Bit stored in "log" field. Thus it would basically duplicate the amount of data saves.

I think an option like preserve_fields logsource,otherimportantfield would solve this neatly.

martin21 commented 7 years ago

Scratch that. I read Fluentd 0.14 has a parser filter plugin on main page. Thus likely makes no sense to implement it here anymore. Link in Fluentd 0.12 documentation redirected to here.