search

Azure / Azure-Sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.
https://azure.microsoft.com/en-us/services/azure-sentinel/
MIT License
4.5k stars 2.95k forks source link

Duplicated logs ingested into Sentinel with OCI (Azure Functions) Data Connector #10863

Open fa-clavis opened 1 month ago

fa-clavis commented 1 month ago

Describe the bug Hello team,

When I was going to model and create some analytics rules for a client, I noticed that there were multiple logs with the same "id_g" field populated into the OCI_Logs_CL table.

This "id_g" field (in OCI logs it's the eventID) is the unique identifier for which every alert receives, so there shouldn't be multiple logs with the same ID in Sentinel. Source: https://docs.oracle.com/en-us/iaas/Content/Audit/Reference/logeventreference.htm

I have installed the default Azure Functions with the ARM template, while I have crosschecked multiple eventIDs in the OCI audit logs, I'm not sure why the same log is being populated 2,3,5, 30 times or more.

See the attached screenshots to understand the issue better.

P.S. I suspected something was odd with the connector because the amount of logs that's being ingested because the amount of logs that are produced in OCI is much less than the client's AWS environment.

To Reproduce Steps to reproduce the behavior:

  1. Install the data connector via ARM: Oracle Cloud Infrastructure (using Azure Functions)
  2. Create all the required backend logging on the OCI side
  3. Insert all the client details in the template and in "CursorType" instead of "group", insert "partition"
  4. Wait for the logs to populate into the table
  5. In the Sentinel Workspace, query the following table "OCI_Logs_CL" and look for the eventIDs in the "id_g" column

Expected behavior The expected behavior is to ingest each eventID once and not multiple times.

Screenshots image image image image image

Additional context

To install and configure the data connector, I used the following resources:

  1. https://docs.oracle.com/en/learn/stream-oci-logs-to-azure-sentinel/index.html#related-links
  2. https://learn.microsoft.com/en-us/azure/sentinel/data-connectors/oracle-cloud-infrastructure

While the log is unique within the Audit Log service, I strongly believe there is something wrong with the Function App or the Python script where the cursor isn't being set up correctly as it pages through the logs.

I believe the problem is somewhere near this line in the main.py file: https://github.com/Azure/Azure-Sentinel/blob/296f272801f36b1945cc2b921d1ec55746617ac4/Solutions/Oracle%20Cloud%20Infrastructure/Data%20Connectors/AzureFunctionOCILogs/main.py#L131

fa-clavis commented 1 month ago

Hello team, Any updates regarding this issue?

v-sudkharat commented 1 month ago

Hey @fa-clavis, Thanks for flagging this issue, we will investigate this issue and get back to you with some updates. Thanks!

fa-clavis commented 1 month ago

Hello @v-sudkharat, any updates regarding this issue?

v-sudkharat commented 1 month ago

Hey @fa-clavis, We are checking the function code for duplicate ingestion of data, just want to know, it's possible to you to share OCI demo account with us? so we can check our changes. Or please let us also know if you have any test workspace into your environment where you can test our change. Thanks!

fa-clavis commented 1 month ago

Hello @v-sudkharat , I talked with the client here, and they don't have a test environment available for OCI, however, they prefer to do the troubleshooting in a quick meet. Is there any way we can schedule this meeting through e-mail or Teams?

v-sudkharat commented 1 month ago

Hi @fa-clavis, let us check with our team to validate some changes, which required the OCI env. once it gets validated will update you. Thanks!

fa-clavis commented 1 month ago

Hello @v-sudkharat, great news! Hope it gets fixed ASAP!

v-sudkharat commented 1 month ago

Hi @fa-clavis, Could you please share few events generated with same id_g with us via a mail - v-sudkharat@microsoft.com to understand the duplication of data in each event. image

v-sudkharat commented 2 weeks ago

Hi @fa-clavis, Just want to check with you, Is there any multiple function app deployed into the environment and point towards the same workspace?

v-sudkharat commented 1 week ago

@fa-clavis, Waiting for your response on above comment, so based on that we can reach out to respective team. Thanks!

v-sudkharat commented 1 week ago

Hi @fa-clavis, Any response for us?

fa-clavis commented 1 week ago

Hello @v-sudkharat, sorry for the delay. No, there is only 1 function app which is now in the "stopped" status.

image

v-sudkharat commented 4 days ago

@fa-clavis, thank you for the update, we are connecting with our concern data connector team for this issue, we will keep you updated. Thanks!