search

Azure / Azure-Sentinel

Cloud-native SIEM for intelligent security analytics for your entire enterprise.
https://azure.microsoft.com/en-us/services/azure-sentinel/
MIT License
4.53k stars 2.97k forks source link

Meraki Syslog Parser not parsing some events #9471

Closed joseff8 closed 4 months ago

joseff8 commented 10 months ago

Is your feature request related to a problem? Please describe. Meraki Syslog Parser not parsing some events

Solutions/CiscoMeraki/Parsers/CiscoMeraki.txt

Describe the solution you'd like Would like the parser to handle additonal "LogType" events like: "firewall" "vpn_firewall" "ip_flow_start" "ip_flow_end"

Additional context Not very good in KQL, however i was able to add the basics: srcipaddr example: LogType has "ip_flow_start", extract(@"src=([0-9.]+)\s",1,Substring), LogType has "ip_flow_end", extract(@"src=([0-9.]+)\s",1,Substring), LogType has "vpn_firewall", extract(@"src=([0-9.]+)\s",1,Substring), LogType has "firewall", extract(@"src=([0-9.]+)\s",1,Substring),

Looks like the extract is the same for "flows" in each case

Added the LogType Events here as will LogType !in ("urls", "airmarshal_events","security_event","ids-alerts", "events", "vpn_firewall", "firewall", "ip_flow_start", "ip_flow_end") and LogType !contains "flows", iif(isempty(LogType), "", LogType),

github-actions[bot] commented 10 months ago

Thank you for submitting an Issue to the Azure Sentinel GitHub repo! You should expect an initial response to your Issue from the team within 5 business days. Note that this response may be delayed during holiday periods. For urgent, production-affecting issues please raise a support ticket via the Azure Portal.

v-muuppugund commented 10 months ago

Hi @joseff8 , Thanks for flagging this issue, we will investigate this issue and get back to you with some updates by 29Nov23. Thanks!

v-sudkharat commented 10 months ago

Hi @joseff8, Sorry for the delay in response. Could you please share more detailed information about the issue? It would be helpful if you could add screenshots of the error you are encountering.

Thanks!

v-sudkharat commented 10 months ago

Hi @joseff8, we are waiting for your response on above comment. thanks!

joseff8 commented 10 months ago

Hi @v-sudkharat, Could you please let me know what more information is needed? As stated the Meraki parser does not parse the events mentioned. "firewall" "vpn_firewall" "ip_flow_start" "ip_flow_end"

I added example changes to how the parser could include the events.

v-muuppugund commented 9 months ago

Hi @joseff8 ,Could you please share sample data in csv format for this parser issue and also screen shot of the error details.

joseff8 commented 9 months ago

Hi @v-muuppugund, Here is a sample.

Dec 15 07:38:10 10.47.129.253 171122330.11122336118 MX_Firewall vpn_firewall src=x.x.x.x dst=x.x.x.x protocol=udp sport=60940 dport=161 pattern: allow all

Dec 15 07:38:10 10.47.255.253 1711223390.202233603 MX_Firewall firewall src=x.x.x.x dst=x.x.x.x mac=00:xx:xx:xx:xx:xx protocol=udp sport=60944 dport=161 pattern: allow (dst x.x.x.x/24 || dst x.x.x.x/24 || (dst x.x.x.x/23 || dst x.x.x.x/24 || dst x.x.x.x/23) || dst x.x.x.x/26) && (src x.x.x.x/24)

Dec 15 07:38:10 10.47.255.253 1112233890.211223390 MX_Firewall ip_flow_start src=x.x.x.x dst=x.x.x.x protocol=udp sport=60944 dport=161 translated_dst_ip=x.x.x.x translated_port=161

As you can see the "type" of log is vpn_firewall, firewall and ip_flow_start I was able to edit the parser to some degree to get these event "Types" parsed as well: See attached file. CiscoMeraki.txt

v-muuppugund commented 9 months ago

Hi @joseff8 ,Thanks ,will check details and do detailed analysis and will get back to you with an update

v-muuppugund commented 9 months ago

Hi @joseff8 ,After Analyzing the above sample data ,we are not parsing the above log types in parser,so its unable to detect,I am unable to replicate the same issue at my environment,Please add LogType has "firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring), LogType has "ip_flow_start", extract(@"pattern\: ([\S\s]+)", 1, Substring), LogType has "ip_flow_end", extract(@"pattern\: ([\S\s]+)", 1, Substring) and this will work ,need check whole logs can you share whole dump in csv format to v-muuppugund@microsoft.com,so will modify the query and share with you.

v-muuppugund commented 9 months ago

Hi @joseff8 ,I have updated the parser attached Updated.Cisco.Meraki.txt

based on data,Could you please check and let me know if any issues

antigrammatik commented 9 months ago

@v-muuppugund That was really good and got most of the parsing working for the flow start and flow end events, thank you for putting this out here. There were a couple columns that weren't working correctly. The source and destination ports needed to be converted to int and the regex for the new fields needed to be updated to get the right value. Here is my updated version. It still needs a bit of work to make it perfect but it's probably 90% of the way there.

Updated Cisco Meraki 12-21-23.txt

v-sudkharat commented 9 months ago

@antigrammatik, thank you for your response.

v-sudkharat commented 9 months ago

Hi @joseff8, Could you please have a look on this comment- https://github.com/Azure/Azure-Sentinel/issues/9471#issuecomment-1866142467 and let us know if your issue gets resolved. Thanks!

v-muuppugund commented 9 months ago

Hi @joseff8 / @antigrammatik , I have cross checked the parser ,above attached is the updated one,Gentle Reminder,Please check and let me know if any issues

v-muuppugund commented 9 months ago

Hi @joseff8 / @antigrammatik ,Genle Reminder,I have cross checked the parser ,above attached is the updated one, Please check and let me know if any issues, Proceeding with the changes, will raise pr

joseff8 commented 9 months ago

@v-muuppugund, Thanks for the update. Sorry for the late reply, i have been on vacation. The code works better. We are still missing the changes for "pattern".

    Pattern = case(
                        isnotempty(column_ifexists("Pattern", "")), column_ifexists("Pattern", ""),
                        LogType has "flows", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "vpn_firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "firewall", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "ip_flow_start", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        LogType has "ip_flow_end", extract(@"pattern\: ([\S\s]+)", 1, Substring),
                        ""
                   ),
v-muuppugund commented 9 months ago

@joseff8 ,Thanks, Will check on these changes ,will work on it and push it.

v-muuppugund commented 9 months ago

@joseff8 ,I have done the changes,testing with large dump of data for any issues as part of standard practice once done,will raise the PR and push it.

v-muuppugund commented 8 months ago

@joseff8 ,Still data ingesting is happening, Will share updates once Testing is completed and pr is pushed.

azvella commented 7 months ago

This really should make it into the production solution parser file. This was a great find and super helpful for me.

abrbon commented 7 months ago

Another issue with this parser is that it does not work for IPv6 addresses.

I fixed this by changing the following lines of code:

LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)

into

LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)

Made the same changes for all "dst=" IP addresses.

And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".

And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:

NetworkProtocol = toupper(NetworkProtocol)

and added the following to parse ICMP type protocol:

ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

v-muuppugund commented 6 months ago

Another issue with this parser is that it does not work for IPv6 addresses.

I fixed this by changing the following lines of code:

LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring)

into

LogType contains "flow", extract(@"src=([^\s]+)",1,Substring)

Made the same changes for all "dst=" IP addresses.

And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow".

And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply:

NetworkProtocol = toupper(NetworkProtocol)

and added the following to parse ICMP type protocol:

ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

v-muuppugund commented 6 months ago

Another issue with this parser is that it does not work for IPv6 addresses. I fixed this by changing the following lines of code: LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring) into LogType contains "flow", extract(@"src=([^\s]+)",1,Substring) Made the same changes for all "dst=" IP addresses. And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow". And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply: NetworkProtocol = toupper(NetworkProtocol) and added the following to parse ICMP type protocol: ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

Will be doing further analysis of Ip4 address ,Will update you post my analysis and changes.

v-muuppugund commented 6 months ago

Another issue with this parser is that it does not work for IPv6 addresses. I fixed this by changing the following lines of code: LogType has "flows", extract(@"src=([0-9\.]+)\s",1,Substring) into LogType contains "flow", extract(@"src=([^\s]+)",1,Substring) Made the same changes for all "dst=" IP addresses. And also it does not parse "ip_flow_start" and "ip_flow_end" LogTypes. I fixed that also by simply changing all "flows" to "flow". And finally i changed the code for "NetworkProtocol" from:

NetworkProtocol = case(NetworkProtocol has "tcp", "TCP",
                                NetworkProtocol has "udp", "UDP",
                                "")

to simply: NetworkProtocol = toupper(NetworkProtocol) and added the following to parse ICMP type protocol: ICMPType = iif(NetworkProtocol contains "icmp", extract(@"type=([^\s]+)",1,Substring), ""),

Sure ,will check and update you

Will be doing further analysis of Ip4 address ,Will update you post my analysis and changes.

Hi @abrbon / @joseff8 ,I have done the requested changes ,working on testing it,will update you

v-muuppugund commented 5 months ago

HI @joseff8 / @abrbon ,just want to update ,earlier data has been lost due to data retention period,so working on testing the new changes done once data ingestion completed

v-rusraut commented 5 months ago

Hi @joseff8, We are still working on data ingestion, we will update you once testing complete. Thanks

v-rusraut commented 5 months ago

Hi @joseff8, We tried to ingest data, but we are not able to ingest data in few tables. Please run below parser in your workspace where data available. CiscoMeraki.txt and share working screen shots with us. Thanks

v-rusraut commented 5 months ago

Hi @joseff8, Please provide update on above comment. Thanks

v-rusraut commented 5 months ago

Hi @joseff8 , Gentle Reminder: We are waiting for your response on this issue. If you still need to keep this issue active, please respond to it in the next 2 days. If we don't receive a response by 13-05-2024 date, we will be closing this issue. Thanks!

abrbon commented 5 months ago

Hi @joseff8 , Gentle Reminder: We are waiting for your response on this issue. If you still need to keep this issue active, please respond to it in the next 2 days. If we don't receive a response by 13-05-2024 date, we will be closing this issue. Thanks!

Hi, I checked and ran the parser on data from our client but unfortunately have not any data in our tables for “Pattern”.

However I still see the same issues I had with this parser like not recognizing IPv6 addresses and not parsing the “ip_flow_start” and “ip_flow_end” logtypes like I mentioned in my earlier posts.

v-rusraut commented 4 months ago

Hi @joseff8, We have tried to resolve this issue in existing parser, but as it is enhancement/feature request in existing parser, so we have noted this feature request and will share this with our respective concern team. Currently we can't provide any ETA for this. but once we get any further info from our team will share update to you. So, closing this issue from GitHub. If you still need support for this issue, feel free to re-open it any time. Thank you for your co-operation.