tagomoris / fluent-plugin-parser

Other
74 stars 23 forks source link

Record's time is replaced with fluentd's current time in spite of using 'time_parse no'. #36

Closed toya256 closed 8 years ago

toya256 commented 8 years ago

I want to parse messege field to divide into some fields. So I use this parser plugin. But record's time is replaced with fluentd's current time in spite of using 'time_parse no'. Is this bug? Or I have misunderstood? How to keep record's time even after parsing?

<match raw.xxx.execdetail>
  type parser
  remove_prefix raw
  format multiline
  format1 /^(?<raw_message>.*<project name="(?<project_name>\w+)".*)$/

  key_name message
  reserve_data yes
  time_parse no
</match>
tagomoris commented 8 years ago

I checked the code of multiline parser, and found that it ignores the specifier not to return current time. So it's an issue of multiline parser, not this plugin. And, I don't recommend to use multiline parser with this plugin. Multiline parser works well for the strongly consistent sequential input (like in_tail). The input data of this plugin is not.

toya256 commented 8 years ago

Thank you for your reply.

This messege field is a raw xml including new lines. So I think should use the multiline format. I tried to directly use a regular expression to the format parameter, but any lines after the first were ignored.

In case of parsing a field including new lines, I'll try to develop a new formatter(or some other solutions like as replacing new lines before parsing).

tagomoris commented 8 years ago

Writing your own parser plugin looks good idea for your case!

toya256 commented 8 years ago

I'll try to develop a new formatter or parser. Thank you so much for your help!