search

fastly / sigsci-splunk-app

Splunk app for Fastly (Signal Sciences)
https://splunkbase.splunk.com/app/3495
MIT License
9 stars 10 forks source link

Timestamp Recognition #2

Closed oohnana closed 5 years ago

oohnana commented 6 years ago

When setting search to All time (real-time), we see events streaming in with OLD timestamps. I've narrowed this down to Splunk extracting the Last-Modified header (when present) on sigsci-requests and not the "timestamp" field. Additionally, at least in our implementation, %f is not needed as shown in your TIME_FORMAT, as we are not getting milliseconds returned, at least not in "timestamp:". Thoughts on how to fix ?

dacoburn commented 6 years ago

I'll take a look at this. Likely I just need to define the field in the inputs.conf to use the proper field.

dacoburn commented 5 years ago

This is resolved in published version 1.0.14 @aspina

I had to add TIME_PREFIX. What was happening was that the Splunk Event was not finding the timestamp entry in the MAX_LOOK_AHEAD size. Adding the TIME_PREFIX tells it exactly where to start looking.

dacoburn commented 5 years ago

Timestamp still was not matching even though there were no more WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (40) characters of event. Defaulting to timestamp of previous event events in the splunkd.log.

Had to do a slight tweak to the timestamp in the props.conf and it seems to be working in my testing now.

TIME_PREFIX = timestamp
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
dacoburn commented 5 years ago

Latest test in my environment showed the timestamp and event time stamp matching now.