Closed Matt-Dee closed 4 years ago
I'm trying to create something that auto generates groks and then dynamically creates a fluentd.conf file based on the discovered groks.
My question is. Can you preserve the original log message that the grok filter parses and add that as a field to your final record?
<source> @type tail path /root/Hadoop-2k tag hadoop_grok_log <parse> @type none </parse> </source> <filter **> @type parser key_name message <parse> @type grok grok_name_key grok_name grok_failure_key grokfailure <grok> name a1946ed9-1075-3216-a812-59dd49140d9b pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[RMCommunicator Allocator\] %{JAVACLASS:javaOne}: Cannot assign container Container: \[ContainerId: .*NodeId: %{HOSTPORT:hostOne}, NodeHttpAddress: %{HOSTPORT:hostTwo}, Resource: <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}>, Priority: .*Token: Token { kind: ContainerToken, service: %{HOSTPORT:hostFive} }, \] for a map as either .*memory less than required <%{HOSTPORT:hostThree}, %{HOSTPORT:hostFour}> or no pending map tasks - maps\.isEmpty=true </grok> <grok> name 1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e pattern %{TIMESTAMP_ISO8601:timeOne} %{LOGLEVEL:loglevelOne} \[IPC Server handler %{POSINT:numberOne} on %{POSINT:numberTwo}\] %{JAVACLASS:javaOne}: Diagnostics report from .*%{LOGLEVEL:loglevelTwo}: %{JAVACLASS:javaTwo}: No Route to Host from .*%{IPV4:hostTwo} to %{HOSTPORT:hostOne} failed on socket timeout exception: %{JAVACLASS:javaTwo}: No route to host: no further information; For more details see: </grok> </parse> </filter> <match **> @type stdout </match>
What is captured:
{"timeOne":"2015-10-18 18:04:10,002","loglevelOne":"INFO","javaOne":"org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator","hostOne":"MSRA-SA-39.fareast.corp.microsoft.com:28345","hostTwo":"MSRA-SA-39.fareast.corp.microsoft.com:8042","hostThree":"memory:1024","hostFour":"vCores:1","hostFive":"172.22.149.145:28345","grok_name":"a1946ed9-1075-3216-a812-59dd49140d9b"} {"timeOne":"2015-10-18 18:06:26,029","loglevelOne":"INFO","numberOne":"13","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"} {"timeOne":"2015-10-18 18:06:28,217","loglevelOne":"INFO","numberOne":"4","numberTwo":"62270","javaOne":"org.apache.hadoop.mapred.TaskAttemptListenerImpl","loglevelTwo":"Error","javaTwo":"java.net.NoRouteToHostException","hostTwo":"127.0.0.1","hostOne":"msra-sa-41:9000","grok_name":"1ebc5ab2-6c6e-34bc-9d39-00d1d60c4a3e"}
What would I add to my configuration to capture the original log message for the parsed logs?
I'm trying to create something that auto generates groks and then dynamically creates a fluentd.conf file based on the discovered groks.
My question is. Can you preserve the original log message that the grok filter parses and add that as a field to your final record?
What is captured:
What would I add to my configuration to capture the original log message for the parsed logs?