search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

Event forwarding to Splunk HEC fails w/ W3C events #77

Open mcatcribl opened 7 months ago

mcatcribl commented 7 months ago

Hi @joeymoore.

I'm working w/ a shop who is outputting events in W3C and spent quite a while trying to figure out why events weren't being sent to Splunk HEC.

We were able to trace it down to the code block '# Loop over the data/messages array and create the relevant object(s) to be sent.'

https://github.com/imperva/incapsula-logs-downloader/blob/0bf80d027746f5d9b19dc3d082be70549765f62b/script/HttpClient.py#L49

Looks like it's expecting a 'start=' that exists in the message body of CEF/LEEF formatted events, but not in W3C.

In our case, it downloaded and wrote the events to disk, but failed to send the event(s) and didn't log any errors.

I don't know that a ton of folks are using W3C instead of CEF/LEEF, but just figured I'd share.

joeymoore commented 7 months ago

You are correct @mcatcribl , I did not account for all log types. Now if there is a need for the W3C log then I will add some code to handle the different log types. Finding the start time in W3C would require some regex but should be too hard. Let me know if you think this is valuable.

mcatcribl commented 7 months ago

Thanks for the reply @joeymoore.. It'd be great if it's not too much trouble!

Unfortunately, this Imperva customer can't change the output from W3C due to downstream dependencies.

Our current work around = pointing a file collector at /export, but it sure would be nice to just set up a HEC destination and avoid the extra complexity.

PS: If it helps, I can extract / calculate the correct timestamp when it lands on our side. But as it stands, it looks like when the parsing of the timestamp fails, it doesn't fall back to the current time and go on sending the message.