search

mfesiem / msiempy

McAfee SIEM API Python wrapper
https://mfesiem.github.io/docs/msiempy/msiempy.html
MIT License
23 stars 9 forks source link

Rule_NDSNormSigID.msg breaks Queries in 11.3.2 #93

Open lzdx2812 opened 4 years ago

lzdx2812 commented 4 years ago

Describe When trying to get Rule_NDSNormSigID.msg, all results break, for example:

Result without asking for Rule_NDSNormSigID.msg field

| LastTime | Rule.msg | | 10/27/2020 14:10:44 | TCP Scan (horizontal) |

Result asking for Rule_NDSNormSigID.msg

| LastTime | Rule.msg | Rule_NDSNormSigID.msg | | 10/27/2020 14:10:44 | Unknown_0 | TCP Scan (horizontal) |

Results are mixed, Rule.msg gents wrong info, Rule_NDSNormSigID.msg contains Rule.msg, Rule_NDSNormSigID.msg shoud contain correct normalization name, however it puts that "unknown_0".

SIEM and msiempy versions:

Additional context Every other query works with every field except Rule_NDSNormSigID.msg

Code Used:

from msiempy import EventManager, FieldFilter print('Simple event query sorted by AlertID') events = EventManager( time_range='PREVIOUS_DAY', fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg'], filters=[ FieldFilter('SrcIP', ['X.X.X.X'], operator='IN'), ], order=(('ASCENDING', 'AlertID')), limit=100) # Will only load 100 events (per query) events.load_data() print(events) print(events.get_text(fields=['LastTime','Rule.msg','Rule_NDSNormSigID.msg']))

tristanlatr commented 4 years ago

Hello,

Thanks for raising this issue, I'll do my best to fix it.

This is strange... The "Unknown_0" value do not come from the msiempy library.

Can you enable verbose and configure a logfile in your config file ? Then paste (carefully) the logs here please so I can investigate !

Thanks!

tristanlatr commented 4 years ago

The more I think about it the more I believe this is not a library issue but more a McAfee issue...

But to be 100% certain of that, please adjust this standalone script to get fields 'LastTime','Rule.msg','Rule_NDSNormSigID.msg' on one side and 'LastTime','Rule.msg' on the other and compare the raw results.

https://github.com/mfesiem/msiem/blob/master/standalone/offsetIssue.py

lzdx2812 commented 4 years ago

Thank you, Here you will see the debug from de Script I used, asking for Rule_NDSNormSigID.msg

2020-10-28 12:45:20,042 - DEBUG - Calling nitro request : event_query kwargs={'time_range': 'PREVIOUS_DAY', 'order_direction': 'ASCENDING', 'order_field': 'AlertID', 'fields': [{'name': 'Rule.msg'}, {'name': 'LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'IPSIDAlertID'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'SrcIP'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '27.156.248.78'}]}], 'limit': 100, 'offset': 0, 'includeTotal': False} 2020-10-28 12:45:20,043 - DEBUG - Calling nitro request : login kwargs= 2020-10-28 12:45:20,044 - DEBUG - Requesting HTTP post login 2020-10-28 12:45:31,158 - DEBUG - Returning raw requests Response object : <Response [201]> 2020-10-28 12:45:31,160 - DEBUG - Unpacking SIEM response: { "privileges" : { "master" : true, "admin" : false, "power" : false, "audit" : false, "crypto" : false, "systemSettings" : { "read" : true, "write" : true }, 2020-10-28 12:45:31,165 - DEBUG - Calling nitro request : build_stamp kwargs={} 2020-10-28 12:45:31,166 - DEBUG - Requesting HTTP post essmgtGetBuildStamp with data None 2020-10-28 12:45:31,473 - DEBUG - Unpacking SIEM response: { "return": { "buildStamp" : "11.3.2 20200730152039" } } 2020-10-28 12:45:31,479 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'buildStamp': '11.3.2 20200730152039'} 2020-10-28 12:45:31,482 - DEBUG - Requesting HTTP post v2/qryExecuteDetail?type=EVENT&reverse=false with data {'config': {'timeRange': 'PREVIOUS_DAY', 'fields': [{'name': 'Rule.msg'}, {'name': 'LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'IPSIDAlertID'}], 'filters': [{'type': 'EsmFieldFilter', 'field': {'name': 'SrcIP'}, 'operator': 'IN', 'values': [{'type': 'EsmBasicValue', 'value': '27.156.248.78'}]}], 'limit': 100, 'offset': 0, 'order': [{'field': {'name': 'AlertID'}, 'direction': 'ASCENDING'}]}} 2020-10-28 12:45:32,629 - DEBUG - Unpacking SIEM response: { "totalRows" : 0, "stopTime" : "10/28/2020 00:00:00", "resultID" : "1224883172", "totalResultID" : "0", "groupByString" : "", "startTime" : "10/27/2020 00:00:00", "countColumn" : 0, " 2020-10-28 12:45:32,635 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'totalRows': 0, 'stopTime': '10/28/2020 00:00:00', 'resultID': '1224883172', 'totalResultID': '0', 'groupByString': '', 'startTime': '10/27/2020 00:00:00', 'countColumn': 0, 'labelColumn': 0, 'attrib 2020-10-28 12:45:32,638 - DEBUG - Waiting for EsmRunningQuery object : {'totalRows': 0, 'stopTime': '10/28/2020 00:00:00', 'resultID': '1224883172', 'totalResultID': '0', 'groupByString': '', 'startTime': '10/27/2020 00:00:00', 'countColumn': 0, 'labelColumn': 0, 'attributeColumn': 0, 'drilldownColumn': 1} 2020-10-28 12:45:32,641 - DEBUG - Waiting for the query to be executed on the SIEM... 2020-10-28 12:45:32,641 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:32,646 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:32,822 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 0, "milliseconds" : 164 } 2020-10-28 12:45:32,826 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 0, 'milliseconds': 164} 2020-10-28 12:45:33,029 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,030 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:33,160 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 0, "milliseconds" : 537 } 2020-10-28 12:45:33,162 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 0, 'milliseconds': 537} 2020-10-28 12:45:33,364 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,365 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:33,576 - DEBUG - Unpacking SIEM response: { "complete" : false, "percentComplete" : 99, "milliseconds" : 907 } 2020-10-28 12:45:33,579 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': False, 'percentComplete': 99, 'milliseconds': 907} 2020-10-28 12:45:33,782 - DEBUG - Calling nitro request : query_status kwargs={'resultID': '1224883172'} 2020-10-28 12:45:33,784 - DEBUG - Requesting HTTP post v2/qryGetStatus with data {'resultID': 1224883172} 2020-10-28 12:45:34,044 - DEBUG - Unpacking SIEM response: { "complete" : true, "percentComplete" : 100, "milliseconds" : 1338 } 2020-10-28 12:45:34,048 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'complete': True, 'percentComplete': 100, 'milliseconds': 1338} 2020-10-28 12:45:34,049 - DEBUG - Calling nitro request : query_result kwargs={'startPos': 0, 'numRows': 100, 'resultID': '1224883172'} 2020-10-28 12:45:34,052 - DEBUG - Requesting HTTP post v2/qryGetResults?startPos=0&numRows=100&reverse=false with data {'resultID': 1224883172} 2020-10-28 12:45:34,536 - DEBUG - Unpacking SIEM response: { "columns" : [ { "name" : "Rule.msg" }, { "name" : "Alert.LastTime" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.IPSIDAlertID" } ], "rows" : [ { "values" 2020-10-28 12:45:34,547 - DEBUG - <Response [200]> -> Result (<class 'dict'>): {'columns': [{'name': 'Rule.msg'}, {'name': 'Alert.LastTime'}, {'name': 'Rule_NDSNormSigID.msg'}, {'name': 'Alert.IPSIDAlertID'}], 'rows': [{'values': ['Internet Policy', '10/27/2020 14:10:44', 'Unkno 2020-10-28 12:45:34,552 - DEBUG - Calling nitro request : close_query kwargs={'resultID': '1224883172'} 2020-10-28 12:45:34,555 - DEBUG - Requesting HTTP post v2/qryClose with data {'resultID': 1224883172} 2020-10-28 12:45:34,796 - DEBUG - Unpacking SIEM response: 2020-10-28 12:45:34,799 - DEBUG - <Response [200]> -> Result (<class 'str'>):

lzdx2812 commented 4 years ago

Here is the output from the standalone script when Rule_NDSNormSigID.msg is NOT asked for (is OK I think):

POSTING / "" POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 0}} POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 50}} POSTING qryGetStatus {"resultID": {"value": 1699380593}} POSTING qryGetStatus {"resultID": {"value": 1721154706}} POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1699380593}} Result #1 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "FortiGate_UTM Attack Event IPS Custom", "27.156.248.78" ] } ] } } POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1721154706}} Result #2 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "FortiGate_UTM Attack Event IPS Custom", "27.156.248.78" ] } ] } }

lzdx2812 commented 4 years ago

Here is the output from the standalone script when Rule_NDSNormSigID.msg is asked for (Not Okay I think)

POSTING / "" POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "Rule_NDSNormSigID.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 0}} POSTING qryExecuteDetail?type=EVENT&reverse=false {"config": {"timeRange": "PREVIOUS_DAY", "fields": [{"name": "LastTime"}, {"name": "Rule.msg"}, {"name": "Rule_NDSNormSigID.msg"}, {"name": "SrcIP"}], "filters": [{"type": "EsmFieldFilter", "field": {"name": "SrcIP"}, "operator": "IN", "values": [{"type": "EsmBasicValue", "value": "27.156.248.78"}]}], "limit": 100, "offset": 50}} POSTING qryGetStatus {"resultID": {"value": 2514887835}} POSTING qryGetStatus {"resultID": {"value": 1483268491}} POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 2514887835}} Result #1 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "Exploit", "Unknown_0", "" ] } ] } } POSTING qryGetResults?startPos=0&numRows=1&reverse=false {"resultID": {"value": 1483268491}} Result #2 { "return": { "columns" : [ { "name" : "Alert.LastTime" }, { "name" : "Rule.msg" }, { "name" : "Rule_NDSNormSigID.msg" }, { "name" : "Alert.SrcIP" } ], "rows" : [ { "values" : [ "10/27/2020 14:10:08", "Exploit", "Unknown_0", "" ] } ] } }

tristanlatr commented 4 years ago

I'm now formal, you have found a bug in the McAfee SIEM ! Now you can open a support ticket to get this fixed :D Congratulation @lzdx2812 !

As a reference this, type of output is the issue. It mixes up everything and totally remove one of the results fields, for you, that was ScrIP, in my case it's IPSIDAlertID.

DEBUG - Unpacking SIEM response: {
  "columns" : [ {
    "name" : "Rule_NDSNormSigID.msg"
  }, {
    "name" : "Rule.msg"
  }, {
    "name" : "Alert.IPSIDAlertID"
  }, {
    "name" : "Alert.LastTime"
  } ],
  "rows" : [ {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:37:52.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:37:33.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:36:58.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:36:32.000", "" ]
  }, {
    "values" : [ "User Logon", "Unknown_0", "10/28/2020 19:35:50.000", "" ]
  }, {
    "values" : [ "Postfix Connect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Disconnect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Disconnect from host", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Client message transfer", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  }, {
    "values" : [ "Postfix Message moved to active queue for delivery", "Unknown_0", "10/28/2020 19:34:18.000", "" ]
  } ]
}

The same query, without Rule_NDSNormSigID.msg field give a normal results:

DEBUG - Unpacking SIEM response: {
  "columns" : [ {
    "name" : "Alert.LastTime"
  }, {
    "name" : "Alert.IPSIDAlertID"
  }, {
    "name" : "Rule.msg"
  } ],
  "rows" : [ {
    "values" : [ "10/28/2020 19:43:13", "144115188075855872|18477", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:42:13", "144115188075855872|18476", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:41:22", "144115188075855872|18475", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:37:52", "144115188075855872|18474", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:37:33", "144115188075855872|18473", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:36:58", "144115188075855872|18472", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:36:32", "144115188075855872|18471", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:35:50", "144115188075855872|18470", "User Logon" ]
  }, {
    "values" : [ "10/28/2020 19:34:18", "144116287604260864|646376", "Postfix Connect from host" ]
  }, {
    "values" : [ "10/28/2020 19:34:18", "144116287604260864|646378", "Postfix Disconnect from host" ]
  } ]
}

I'm using SIEM 11.3.0

tristanlatr commented 4 years ago

I'll keep this issue open. Please keep us posted about the status of your support ticket ! :)

lzdx2812 commented 4 years ago

will do, thank you!

tristanlatr commented 3 years ago

Any news @lzdx2812 ?

Thanks

lzdx2812 commented 3 years ago

Hi!, support was not helpful at all about this, they simply acknowledge the error but did no more.