fluent / fluent-plugin-grok-parser

Fluentd's Grok parser
Other
107 stars 31 forks source link

Question: Can you preserve original log message? #82

Closed Matt-Dee closed 4 years ago

Matt-Dee commented 4 years ago

I'm trying to create something that auto generates groks and then dynamically creates a fluentd.conf file based on the discovered groks.

My question is. Can you preserve the original log message that the grok filter parses and add that as a field to your final record?

<source>
  @type tail
  path /root/Hadoop-2k
  tag hadoop_grok_log
  <parse>
    @type none
  </parse>
</source> 

<filter **>
    @type parser
    key_name message
    <parse>
        @type grok
        grok_name_key grok_name
        grok_failure_key grokfailure

        <grok>
          name a1946ed9-1075-3216-a812-59dd49140d9b
          pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[RMCommunicator Allocator\] %{JAVACLASS:javaOne}: Cannot assign container Container: \[ContainerId: .*NodeId: %{HOSTPORT:hostOne}, NodeHttpAddress: %{HOSTPORT:hostTwo}, Resource: <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}>, Priority: .*Token: Token { kind: ContainerToken, service: %{HOSTPORT:hostFive} }, \] for a map as either .*memory less than required <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}> or no pending map tasks - maps\.isEmpty=true
         </grok>

        <grok>
          name 1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e
          pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[IPC Server handler %{POSINT:numberOne} on %{POSINT:numberTwo}\] %{JAVACLASS:javaOne}: Diagnostics report from .*%{LOGLEVEL:loglevelTwo}: %{JAVACLASS:javaTwo}: No Route to Host from .*%{IPV4:hostTwo} to %{HOSTPORT:hostOne} failed on socket timeout exception: %{JAVACLASS:javaTwo}: No route to host: no further information; For more details see:
        </grok>
    </parse>
</filter>

<match **>
  @type stdout
</match>

What is captured:

{"timeOne":"2015-10-18 18:04:10,002","loglevelOne":"INFO","javaOne":"org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator","hostOne":"MSRA-SA-39.fareast.corp.microsoft.com:28345","hostTwo":"MSRA-SA-39.fareast.corp.microsoft.com:8042","hostThree":"memory:1024","hostFour":"vCores:1","hostFive":"172.22.149.145:28345","grok_name":"a1946ed9-1075-3216-a812-59dd49140d9b"}
{"timeOne":"2015-10-18 18:06:26,029","loglevelOne":"INFO","numberOne":"13","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}
 {"timeOne":"2015-10-18 18:06:28,217","loglevelOne":"INFO","numberOne":"4","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}

What would I add to my configuration to capture the original log message for the parsed logs?