Open lzdx2812 opened 4 years ago
Hello,
Thanks for raising this issue, I'll do my best to fix it.
This is strange... The "Unknown_0" value do not come from the msiempy library.
Can you enable verbose
and configure a logfile
in your config file ?
Then paste (carefully) the logs here please so I can investigate !
Thanks!
The more I think about it the more I believe this is not a library issue but more a McAfee issue...
But to be 100% certain of that, please adjust this standalone script to get fields 'LastTime','Rule.msg','Rule_NDSNormSigID.msg' on one side and 'LastTime','Rule.msg' on the other and compare the raw results.
https://github.com/mfesiem/msiem/blob/master/standalone/offsetIssue.py
Thank you, Here you will see the debug from de Script I used, asking for Rule_NDSNormSigID.msg
2020-10-28 12:45:20,042 - DEBUG - Calling nitro request : event_query kwargs={'time_range': 'PREVIOUS_DAY', 'order_direction': 'ASCENDING', 'order_field': 'AlertID', 'fields': [{'name': 'Rule.msg'}, {'name': 'LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'IPSIDAlertID'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'SrcIP'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '27.156.248.78'}]}], 'limit': 100, 'offset': 0, 'includeTotal': False} 2020-10-28 12:45:20,043 - DEBUG - Calling nitro request : login kwargs= 2020-10-28 12:45:20,044 - DEBUG - Requesting HTTP post login 2020-10-28 12:45:31,158 - DEBUG - Returning raw requests Response object : <Response [201]> 2020-10-28 12:45:31,160 - DEBUG - Unpacking SIEM response: { "privileges" : { "master" : true, "admin" : false, "power" : false, "audit" : false, "crypto" : false, "systemSettings" : { "read" : true, "write" : true }, 2020-10-28 12:45:31,165 - DEBUG - Calling nitro request : build_stamp kwargs={} 2020-10-28 12:45:31,166 - DEBUG - Requesting HTTP post essmgtGetBuildStamp with data None 2020-10-28 12:45:31,473 - DEBUG - Unpacking SIEM response: { "return": { "buildStamp" : "11.3.2 20200730152039" } } 2020-10-28 12:45:31,479 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'buildStamp': '11.3.2 20200730152039'} 2020-10-28 12:45:31,482 - DEBUG - Requesting HTTP post v2/qryExecuteDetail?type=EVENT&reverse=false with data {'config': {'timeRange': 'PREVIOUS_DAY', 'fields': [{'name': 'Rule.msg'}, {'name': 'LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'IPSIDAlertID'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'SrcIP'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '27.156.248.78'}]}], 'limit': 100, 'offset': 0, 'order': [{'field': {'name': 'AlertID'}, 'direction': 'ASCENDING'}]}} 2020-10-28 12:45:32,629 - DEBUG - Unpacking SIEM response: { "totalRows" : 0, "stopTime" : "10/28/2020 00:00:00", "resultID" : "1224883172", "totalResultID" : "0", "groupByString" : "", "startTime" : "10/27/2020 00:00:00", "countColumn" : 0, " 2020-10-28 12:45:32,635 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'totalRows': 0, 'stopTime': '10/28/2020 00:00:00', 'resultID': '1224883172', 'totalResultID': '0', 'groupByString': '', 'startTime': '10/27/2020 00:00:00', 'countColumn': 0, 'labelColumn': 0, 'attrib 2020-10-28 12:45:32,638 - DEBUG - Waiting for EsmRunningQuery object : {'totalRows': 0, 'stopTime': '10/28/2020 00:00:00', 'resultID': '1224883172', 'totalResultID': '0', 'groupByString': '', 'startTime': '10/27/2020 00:00:00', 'countColumn': 0, 'labelColumn': 0, 'attributeColumn': 0, 'drilldownColumn': 1} 2020-10-28 12:45:32,641 - DEBUG - Waiting for the query to be executed on the SIEM... 2020-10-28 12:45:32,641 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:32,646 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:32,822 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 0, "milliseconds" : 164 } 2020-10-28 12:45:32,826 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 0, 'milliseconds': 164} 2020-10-28 12:45:33,029 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,030 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:33,160 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 0, "milliseconds" : 537 } 2020-10-28 12:45:33,162 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 0, 'milliseconds': 537} 2020-10-28 12:45:33,364 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,365 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:33,576 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 99, "milliseconds" : 907 } 2020-10-28 12:45:33,579 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 99, 'milliseconds': 907} 2020-10-28 12:45:33,782 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,784 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:34,044 - DEBUG - Unpacking SIEM response: { "complete" : true, "percentComplete" : 100, "milliseconds" : 1338 } 2020-10-28 12:45:34,048 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': True, 'percentComplete': 100, 'milliseconds': 1338} 2020-10-28 12:45:34,049 - DEBUG - Calling nitro request : query_result kwargs={'startPos': 0, 'numRows': 100, 'resultID': '1224883172'} 2020-10-28 12:45:34,052 - DEBUG - Requesting HTTP post v2/qryGetResults?startPos=0&numRows=100&reverse=false with data {'resultID': 1224883172} 2020-10-28 12:45:34,536 - DEBUG - Unpacking SIEM response: { "columns" : [ { "name" : "Rule.msg" }, { "name" : "Alert.LastTime" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.IPSIDAlertID" } ], "rows" : [ { "values" 2020-10-28 12:45:34,547 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'columns': [{'name': 'Rule.msg'}, {'name': 'Alert.LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'Alert.IPSIDAlertID'}], 'rows': [{'values': ['Internet Policy', '10/27/2020 14:10:44', 'Unkno 2020-10-28 12:45:34,552 - DEBUG - Calling nitro request : close_query kwargs={'resultID': '1224883172'} 2020-10-28 12:45:34,555 - DEBUG - Requesting HTTP post v2/qryClose with data {'resultID': 1224883172} 2020-10-28 12:45:34,796 - DEBUG - Unpacking SIEM response: 2020-10-28 12:45:34,799 - DEBUG - <Response [200]> -> Result (<class 'str'>):
Here is the output from the standalone script when Rule_NDSNormSigID.msg is NOT asked for (is OK I think):
POSTING / "" POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 0}} POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 50}} POSTING qryGetStatus {"resultID": {"value": 1699380593}} POSTING qryGetStatus {"resultID": {"value": 1721154706}} POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1699380593}} Result #1 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "FortiGate_UTM Attack Event IPS Custom", "27.156.248.78" ] } ] } } POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1721154706}} Result #2 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "FortiGate_UTM Attack Event IPS Custom", "27.156.248.78" ] } ] } }
Here is the output from the standalone script when Rule_NDSNormSigID.msg is asked for (Not Okay I think)
POSTING / "" POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "Rule_NDSNormSigID.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 0}} POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "Rule_NDSNormSigID.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 50}} POSTING qryGetStatus {"resultID": {"value": 2514887835}} POSTING qryGetStatus {"resultID": {"value": 1483268491}} POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 2514887835}} Result #1 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "Exploit", "Unknown_0", "" ] } ] } } POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1483268491}} Result #2 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "Exploit", "Unknown_0", "" ] } ] } }
I'm now formal, you have found a bug in the McAfee SIEM ! Now you can open a support ticket to get this fixed :D Congratulation @lzdx2812 !
As a reference this, type of output is the issue. It mixes up everything and totally remove one of the results fields, for you, that was ScrIP, in my case it's IPSIDAlertID.
DEBUG - Unpacking SIEM response: {
"columns" : [ {
"name" : "Rule_NDSNormSigID.msg"
}, {
"name" : "Rule.msg"
}, {
"name" : "Alert.IPSIDAlertID"
}, {
"name" : "Alert.LastTime"
} ],
"rows" : [ {
"values" : [ "User Logon", "Unknown_0", "10/28/2020 19:37:52.000", "" ]
}, {
"values" : [ "User Logon", "Unknown_0", "10/28/2020 19:37:33.000", "" ]
}, {
"values" : [ "User Logon", "Unknown_0", "10/28/2020 19:36:58.000", "" ]
}, {
"values" : [ "User Logon", "Unknown_0", "10/28/2020 19:36:32.000", "" ]
}, {
"values" : [ "User Logon", "Unknown_0", "10/28/2020 19:35:50.000", "" ]
}, {
"values" : [ "Postfix Connect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
}, {
"values" : [ "Postfix Disconnect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
}, {
"values" : [ "Postfix Disconnect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
}, {
"values" : [ "Postfix Client message transfer", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
}, {
"values" : [ "Postfix Message moved to active queue for delivery", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
} ]
}
The same query, without Rule_NDSNormSigID.msg
field give a normal results:
DEBUG - Unpacking SIEM response: {
"columns" : [ {
"name" : "Alert.LastTime"
}, {
"name" : "Alert.IPSIDAlertID"
}, {
"name" : "Rule.msg"
} ],
"rows" : [ {
"values" : [ "10/28/2020 19:43:13", "144115188075855872|18477", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:42:13", "144115188075855872|18476", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:41:22", "144115188075855872|18475", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:37:52", "144115188075855872|18474", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:37:33", "144115188075855872|18473", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:36:58", "144115188075855872|18472", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:36:32", "144115188075855872|18471", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:35:50", "144115188075855872|18470", "User Logon" ]
}, {
"values" : [ "10/28/2020 19:34:18", "144116287604260864|646376", "Postfix Connect from host" ]
}, {
"values" : [ "10/28/2020 19:34:18", "144116287604260864|646378", "Postfix Disconnect from host" ]
} ]
}
I'm using SIEM 11.3.0
I'll keep this issue open. Please keep us posted about the status of your support ticket ! :)
will do, thank you!
Any news @lzdx2812 ?
Thanks
Hi!, support was not helpful at all about this, they simply acknowledge the error but did no more.
Describe When trying to get Rule_NDSNormSigID.msg, all results break, for example:
Result without asking for Rule_NDSNormSigID.msg field
| LastTime | Rule.msg | | 10/27/2020 14:10:44 | TCP Scan (horizontal) |
Result asking for Rule_NDSNormSigID.msg
| LastTime | Rule.msg | Rule_NDSNormSigID.msg | | 10/27/2020 14:10:44 | Unknown_0 | TCP Scan (horizontal) |
Results are mixed, Rule.msg gents wrong info, Rule_NDSNormSigID.msg contains Rule.msg, Rule_NDSNormSigID.msg shoud contain correct normalization name, however it puts that "unknown_0".
SIEM and msiempy versions:
Additional context Every other query works with every field except Rule_NDSNormSigID.msg
Code Used:
from msiempy import EventManager, FieldFilter print('Simple event query sorted by AlertID') events = EventManager( time_range='PREVIOUS_DAY', fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg'], filters=[ FieldFilter('SrcIP', ['X.X.X.X'], operator='IN'), ], order=(('ASCENDING', 'AlertID')), limit=100) # Will only load 100 events (per query) events.load_data() print(events) print(events.get_text(fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg']))