Add delay in the data collection to avoid missing events issue.

sophos / Sophos-Central-SIEM-Integration

Simple integration script for 3rd party systems such as SIEMs. Offers command line, file or syslog output in CEF, JSON or key-value pair formats.

121 stars 70 forks source link

Add delay in the data collection to avoid missing events issue. #69

Closed VatsalJagani closed 2 years ago

VatsalJagani commented 2 years ago

Adding extra delay while collecting the events/alerts from Sophos central API. The delay is configurable through the config file. The default is 60 seconds.

Story:

I was seeing many events being missed on Splunk so I started investigating the issue and it seems when I tried to execute the script again I was seeing the events.
So, I think this behavior could be due to some specific host being ahead in time for a few seconds, and Sophos Central would consider events received from that host as a checkpoint when returning the response.
Hence you may miss some events from other hosts which are behind in time from that host which registered the new checkpoint.

anil-sophos commented 2 years ago

Thanks, @VatsalJagani For the suggestion, We incorporated these changes and waiting on the customer confirmation to this change fix this issue or not. will raise the PR once we get this confirmation from the customers who raised the issue.

In your changes, you mimic the cursor value in the make_token_request method. we also need to do the same changes in the make_credentials_request method as well.

VatsalJagani commented 2 years ago

In your changes, you mimic the cursor value in the make_token_request method. we also need to do the same changes in the make_credentials_request method as well.

Yes, that is correct. But I don't have any way to test that part of the code so you can help me update this PR to make those changes if you can. Thanks @anil-sophos

anil-sophos commented 2 years ago

@VatsalJagani Below attached both updated method screenshots. soon will raise this PR. You can update the code based on screenshots and test the things with token configuration. for the make_credentials_request test, you need to configure client_id and client_secret.

VatsalJagani commented 2 years ago

Thanks @anil-sophos. I've done the code change, but I'm doing some testing. I'll update the PR as soon as I'm done with the testing.

VatsalJagani commented 2 years ago

@anil-sophos

Thanks for advising on how to test the second method and code change.
After applying code change I was not able to generate the state (siem_sophos.json) file. I was able to figure out in both methods I had to do self.state.save_state outside the while as after break it was not being executed.
After the code change, I've tested the code with both URI and ClientID-ClientSecret and both methods are working fine as expected.
I've updated PR with the code change. Kindly review.

VatsalJagani commented 2 years ago

@anil-sophos - Do you have any information on the timeline for this?

anil-sophos commented 2 years ago

@VatsalJagani We will planning to fix the cursor things on the backend side and on the python script side we will pass collection_delay as a query param.

anil-sophos commented 2 years ago

We have incorporated the collection delay configuration in API. so no need to mimic the cursor value in the script. we provided this feature as a config parameter. Here is the PR Link: https://github.com/sophos/Sophos-Central-SIEM-Integration/pull/77