search

imperva / incapsula-logs-downloader

A Python script for downloading log files from Incapsula
MIT License
30 stars 35 forks source link

SYSLOG sending from script found massively duplicated on SIEM #20

Closed zhinobu38 closed 3 years ago

zhinobu38 commented 3 years ago

Hi,

As my config use the script to send SYSLOG event over to SIEM. I found massive duplication of events on SIEM. both UDP and TCP protocol results the same. Multiple of events are of the same deviceExternalId (Incapsula request id) which mean they are actually the same event. However when I check on the downloaded logfile it has no duplicate at all, which could be the fault at SYSLOG handling stage in the script.

Has anyone experience this before? and how to resolve it?

Thanks.

AaronSeibert commented 3 years ago

Hi @zhinobu38 -

This is likely due to the issues you were having related to #19. The syslog portion processes a log file line by line - however, if you weren't able to process the whole file it wouldn't have added to the past files list - so it would have reprocessed a number of events each time you ran the script until you were able to successfully process the entire log file.

I'd suggest looking at a more recent log file to reconcile and see if you're still getting duplicated events.

zhinobu38 commented 3 years ago

Hi @AaronSeibert ,

I check the downloaded logfile and pretty sure that there is no duplication request written in it. And the issue that was in #19 has been solved already, there is no error at the moment. As the Settings.confg the SYSLOG is enabled to send the event over to SIEM after the logfile is written locally, I'm not sure this would be issue at sending SYSLOG or stream handling state that sending duplicated event over SYSLOG?

AaronSeibert commented 3 years ago

Hi @zhinobu38 -

I'm sorry I wasn't clear - I don't think the duplications would have been in the downloaded log file. I believe that the failures you were experiencing for #19 caused the same log file to be processed multiple times during troubleshooting. A quick indication would be to see if you are still receiving duplicated events in new events that have been processed since you solved #19.

zhinobu38 commented 3 years ago

Hi @AaronSeibert

As tested recently, The duplication of syslog event is still happen. And there is no error of .py terminal. I test with both TCP and UDP, both result the same.

zildjiean commented 3 years ago

Hi,

I have the same issue.

zhinobu38 commented 3 years ago

Hi @zildjiean ,

I still cannot find the solution for this yet. May i know which OS and Python version you use? What I have been tested so far is :

TonPiyapong commented 3 years ago

Hi @zhinobu38 ,

I have submitted a pull request already, If you need to change anything, please let me know.

23 <--

AaronSeibert commented 3 years ago

I've merged PR #23 please update and validate