jfrog / log-analytics-splunk

JFrog Splunk Log Analytics Integration
Apache License 2.0
9 stars 9 forks source link

fluent.conf.rt log parsing issue #28

Open garegintamrazyan07 opened 3 years ago

garegintamrazyan07 commented 3 years ago

in https://github.com/jfrog/log-analytics-splunk/blob/master/fluent.conf.rt line 111

expression ^(?<timestamp>[^ ]*)\|(?<trace_id>[^\|]*)\|(?<remote_address>[^\|]*)\|(?<username>[^\|]*)\|(?<request_method>[^\|]*)\|(?<request_url>[^\|]*)\|(?<return_status>[^\|]*)\|(?<response_content_length>[^\|]*)\|(?<request_content_length>[^\|]*)\|(?<request_duration>[^\|]*)\|(?<request_user_agent>.+)$

response_content_length comes first before request_content_length

As per https://www.jfrog.com/confluence/display/JFROG/Logging request log format looks like this:

Timestamp | Trace ID | Remote Address | Username | Request method | Request URL | Return Status | Request Content Length | Response Content Length | Request Duration | Request User Agent

Request Content Length comes first, which means request_content_length should come first as well in fluent.conf.rt

mikeycmccarthy commented 3 years ago

I'm also seeing issues with the example given within these repos, not sure if this was the error string you are seeing:

2021-10-05 10:41:36 +0000 [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="pattern not matched with data '2021-10-05T10:41:36.245Z [jffe ] [\e[34M[INFO ]\e[39M] [                ] [                              ] [main                ] - attempting pinging artifactory for 180 retires and 1.0s interval for total of 3 minutes'" location=nil tag="jfrog.rt.frontend.service" time=2021-10-05 10:41:36.272250853 +0000 record={"message"=>"2021-10-05T10:41:36.245Z [jffe ] [\e[34M[INFO ]\e[39M] [                ] [                              ] [main                ] - attempting pinging artifactory for 180 retires and 1.0s interval for total of 3 minutes"}

I'll make the changes you suggest to a copy of this fluentd.conf.rt and report back if I have more luck.