search

Azure / Azure-Sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.
https://azure.microsoft.com/en-us/services/azure-sentinel/
MIT License
4.58k stars 3k forks source link

DomainEntity_EmailUrlInfo TI detection creates memory issues on large data sets #11340

Open MSJosh opened 4 days ago

MSJosh commented 4 days ago

KQL Query for DomainEntity_EmailUrlInfo is not optimized for larger data sets leading to memory issues in LAW.

Source- https://github.com/Azure/Azure-Sentinel/blob/master/Solutions/Threat%20Intelligence/Analytic%20Rules/DomainEntity_EmailUrlInfo.yaml

To Reproduce Run Existing query against data set that is TB a day in email traffic. Analysts will get excessive memory alerts and in cases for larger deployments SEM00023 errors as materialize exceeds the limit and fails the query. Image

Expected behavior Utilize improved query performance by dropping materialize and split data up more to improve performance.

Here is a fix which improves overall performance of query by over 90 percent on same data sets with same results.

https://github.com/MSJosh/documentation/blob/main/Sentinel/Misc.%20KQL/TI/DomainEntity_EmailUrlInfo.yaml

v-rusraut commented 2 days ago

Hi @MSJosh , thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks!