ait-aecid / logdata-anomaly-miner

This tool parses log data and allows to define analysis pipelines for anomaly detection. It was designed to run the analysis with limited resources and lowest possible permissions to make it suitable for production server use.
GNU General Public License v3.0
70 stars 23 forks source link

\n in JSON string leads to unparsed event #1315

Closed landauermax closed 3 months ago

landauermax commented 5 months ago

It is possible to parse the following event

{"@timestamp": "2024-04-18T11:00:42.606Z", "message": "abc", "winlog": {"user": {"name": "SYSTEM", "type": "Well Known Group", "identifier": "S-1-5-18", "domain": "NT AUTHORITY"}, "process": {"pid": 2316, "thread": {"id": 3928}}, "channel": "Microsoft-Windows-Sysmon/Operational", "computer_name": "atb-client02.aecid-testbed.com", "api": "wineventlog", "provider_name": "Microsoft-Windows-Sysmon", "opcode": "Info", "provider_guid": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}", "event_data": {"TargetUser": "AECID-TESTBED\\Bob", "SourceProcessId": "3396", "TargetImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "GrantedAccess": "0x1fffff", "SourceProcessGUID": "{816bf8e6-f731-6620-000f-000000000900}", "TargetProcessId": "1340", "SourceThreadId": "6960", "CallTrace": "C:\\windows\\SYSTEM32\\ntdll.dll+9e8f4|C:\\windows\\System32\\KERNELBASE.dll+58745|C:\\windows\\System32\\KERNELBASE.dll+54b93|C:\\windows\\System32\\ADVAPI32.dll+17d20|C:\\Program Files\\Mozilla Firefox\\firefox.exe+300bc|C:\\Program Files\\Mozilla Firefox\\firefox.exe+14228|C:\\Program Files\\Mozilla Firefox\\xul.dll+2619da3|C:\\Program Files\\Mozilla Firefox\\xul.dll+a62338|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f27f|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f53e|C:\\Program Files\\Mozilla Firefox\\xul.dll+faf3d|C:\\Program Files\\Mozilla Firefox\\xul.dll+188439e|C:\\Program Files\\Mozilla Firefox\\xul.dll+18834c3|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b51f|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b068|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4f25|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4dd1|C:\\windows\\System32\\ucrtbase.dll+21bb2|C:\\windows\\System32\\KERNEL32.DLL+17344|C:\\Program Files\\Mozilla Firefox\\mozglue.dll+da58|C:\\windows\\SYSTEM32\\ntdll.dll+526b1", "SourceUser": "AECID-TESTBED\\Bob", "RuleName": "-", "UtcTime": "2024-04-18 11:00:42.565", "SourceImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "TargetProcessGUID": "{816bf8e6-fd5a-6620-a50f-000000000900}"}, "record_id": 1403233, "task": "Process accessed (rule: ProcessAccess)", "version": 3, "event_id": "10"}}

but not this event where "message": "\n"

{"@timestamp": "2024-04-18T11:00:42.606Z", "message": "\n", "winlog": {"user": {"name": "SYSTEM", "type": "Well Known Group", "identifier": "S-1-5-18", "domain": "NT AUTHORITY"}, "process": {"pid": 2316, "thread": {"id": 3928}}, "channel": "Microsoft-Windows-Sysmon/Operational", "computer_name": "atb-client02.aecid-testbed.com", "api": "wineventlog", "provider_name": "Microsoft-Windows-Sysmon", "opcode": "Info", "provider_guid": "{5770385f-c22a-43e0-bf4c-06f5698ffbd9}", "event_data": {"TargetUser": "AECID-TESTBED\\Bob", "SourceProcessId": "3396", "TargetImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "GrantedAccess": "0x1fffff", "SourceProcessGUID": "{816bf8e6-f731-6620-000f-000000000900}", "TargetProcessId": "1340", "SourceThreadId": "6960", "CallTrace": "C:\\windows\\SYSTEM32\\ntdll.dll+9e8f4|C:\\windows\\System32\\KERNELBASE.dll+58745|C:\\windows\\System32\\KERNELBASE.dll+54b93|C:\\windows\\System32\\ADVAPI32.dll+17d20|C:\\Program Files\\Mozilla Firefox\\firefox.exe+300bc|C:\\Program Files\\Mozilla Firefox\\firefox.exe+14228|C:\\Program Files\\Mozilla Firefox\\xul.dll+2619da3|C:\\Program Files\\Mozilla Firefox\\xul.dll+a62338|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f27f|C:\\Program Files\\Mozilla Firefox\\xul.dll+a5f53e|C:\\Program Files\\Mozilla Firefox\\xul.dll+faf3d|C:\\Program Files\\Mozilla Firefox\\xul.dll+188439e|C:\\Program Files\\Mozilla Firefox\\xul.dll+18834c3|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b51f|C:\\Program Files\\Mozilla Firefox\\xul.dll+3b068|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4f25|C:\\Program Files\\Mozilla Firefox\\nss3.dll+4dd1|C:\\windows\\System32\\ucrtbase.dll+21bb2|C:\\windows\\System32\\KERNEL32.DLL+17344|C:\\Program Files\\Mozilla Firefox\\mozglue.dll+da58|C:\\windows\\SYSTEM32\\ntdll.dll+526b1", "SourceUser": "AECID-TESTBED\\Bob", "RuleName": "-", "UtcTime": "2024-04-18 11:00:42.565", "SourceImage": "C:\\Program Files\\Mozilla Firefox\\firefox.exe", "TargetProcessGUID": "{816bf8e6-fd5a-6620-a50f-000000000900}"}, "record_id": 1403233, "task": "Process accessed (rule: ProcessAccess)", "version": 3, "event_id": "10"}}

with the following config:

LearnMode: True

LogResourceList:
  - 'file:///home/ubuntu/windows/tmp.log'

Parser:
  - id: event_id
    type: VariableByteDataModelElement
    name: 'event_id'
    args: '0123456789'

  - id: computer_name
    type: AnyByteDataModelElement
    name: 'computer_name'

  - id: ts
    type: DateTimeModelElement
    name: 'timestamp'
    date_format: "%Y-%m-%dT%H:%M:%S.%fZ" # 2024-03-21T20:46:06.955Z

  - id: target_image
    type: AnyByteDataModelElement
    name: 'target_image'

  - id: source_image
    type: AnyByteDataModelElement
    name: 'source_image'

  - id: json
    start: True
    type: JsonModelElement
    name: 'model'
    allow_all_fields: True
    optional_key_prefix: "*"
    key_parser_dict:
      "@timestamp": ts
      winlog:
        computer_name: computer_name

Input:
        timestamp_paths: "/model/@timestamp/timestamp"

Analysis:
   - type: ParserCount
     id: ParserCount
     output_event_handlers:
       - "stpe"
     report_interval: 5

EventHandlers:
    - id: "stpe"
      type: "StreamPrinterEventHandler"
      json: True

    - id: 'stpefile'
      type: 'StreamPrinterEventHandler'
      json: true
      pretty: false
      output_file_path: '/tmp/anomalies.txt'

This is not expected since \n in strings of a JSON event should also be parsed with the AnyByteDataModelElement.

ernstleierzopf commented 4 months ago

This issue happens because special characters like \n or \t are not escaped, because they should rather be interpreted. However, these characters need to be escaped within string values of the json object.

This took quite a long time to debug..