search

elastic / integrations

Elastic Integrations
https://www.elastic.co/integrations
Other
194 stars 419 forks source link

[Windows system.security] IPv6 gsub causes invalid IPs #9650

Closed ethack closed 2 days ago

ethack commented 4 months ago

The following processor can turn valid IPv6 address into invalid ones.

https://github.com/elastic/integrations/blob/af36780d5654792f68466c42ddeb95627e464d17/packages/system/data_stream/security/elasticsearch/ingest_pipeline/default.yml#L11-L15

Examples:

Which then later causes errors such as 'fe80ffff:ffff:ffff' is not an IP string literal. in other processors. It also causes other issues. image

I believe this processor's intent is to transform IPs such as ::ffff:192.168.1.1 into 192.168.1.1. In which case, the pattern should be ^::ffff: to anchor the pattern at the beginning of the string. (See my below comment for a better proposed solution.)

  - gsub:
      field: source.ip
      pattern: "^::ffff:"
      replacement: ""
      ignore_missing: true

I tested this in a simulated pipeline using both fe80::ffff:ffff:fffe and ::ffff:192.168.1.1 values. It correctly left the former untouched and correctly stripped the prefix from the latter.

References:

elasticmachine commented 4 months ago

Pinging @elastic/sec-windows-platform (Team:Security-Windows Platform)

ethack commented 4 months ago

I just uncovered a few additional details.

LaZyDK commented 1 month ago

This is giving a lot of errors for us.

In System integration 1.60.2:

{
    "gsub": {
      "field": "source.ip",
      "pattern": "::ffff:",
      "replacement": "",
      "ignore_missing": true
    }
  }

Logstash output:

{
  "response": {
    "create": {
      "error": {
        "reason": "[1:1203] failed to parse field [source.ip] of type [ip] in document with id 'KLzoLJEBQHuHIPLF3fpq'. Preview of field's value: 'fe80ffff:ffff:ffff'",
        "caused_by": {
          "reason": "'fe80ffff:ffff:ffff' is not an IP string literal.",
          "type": "illegal_argument_exception"
        },
        "type": "document_parsing_exception"
      },
      "status": 400
    }
  },
  "action": [
    "create",
    "{_index=winlogbeat-default}",
    "{agent={name=windows, id=9197309a-3f83-4ec2-bc54-0bb79783b621, type=winlogbeat, ephemeral_id=238e0a1e-ac6f-4451-9ba3-d38f6dd1cdb5, version=8.8.0}, winlog={computer_name=windows.default.example.com, process={pid=4, thread={id=3660}}, keywords=[Audit Failure], channel=Security, event_data={SourceAddress=fe80::ffff:ffff:ffff:ffff, LayerName=%%14610, DestPort=130, RemoteMachineID=S-1-0-0, FilterRTID=95023, SourcePort=0, RemoteUserID=S-1-0-0, Direction=%%14592, LayerRTID=46, ProcessID=4, Protocol=58, DestAddress=ff02::1, Application=System}, opcode=Info, version=1, record_id=184294, task=Filtering Platform Connection, event_id=5157, provider_guid={54849625-5478-4994-a5ba-3e3b0328c30d}, api=wineventlog, provider_name=Microsoft-Windows-Security-Auditing}, log={level=information}, message=The Windows Filtering Platform has blocked a connection.\n\nApplication Information:\n\tProcess ID:\t\t4\n\tApplication Name:\tSystem\n\nNetwork Information:\n\tDirection:\t\tInbound\n\tSource Address:\t\tfe80::ffff:ffff:ffff:ffff\n\tSource Port:\t\t0\n\tDestination Address:\tff02::1\n\tDestination Port:\t\t130\n\tProtocol:\t\t58\n\nFilter Information:\n\tFilter Run-Time ID:\t95023\n\tLayer Name:\t\tReceive/Accept\n\tLayer Run-Time ID:\t46, type=beats, tags=[beats], @timestamp=2024-07-31T09:55:12.012Z, ecs={version=8.0.0}, @version=1, host={hostname=windows, os={build=17763.5576, kernel=10.0.17763.5576 (WinBuild.160101.0800), name=Windows Server 2019 Standard, type=windows, family=windows, version=10.0, platform=windows}, ip=[fe80::168f:f4ec:e986:0000, 10.10.10.10], name=windows, id=42569218-7bbd-4008-83cd-c2fafe8c8a5c, mac=[00-50-FF-FF-FF-FF], architecture=x86_64}, event={code=5157, original=<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-a5ba-3e3b0328c30d}'/><EventID>5157</EventID><Version>1</Version><Level>0</Level><Task>12810</Task><Opcode>0</Opcode><Keywords>0x8010000000000000</Keywords><TimeCreated SystemTime='2024-07-31T09:55:12.012388900Z'/><EventRecordID>184294</EventRecordID><Correlation/><Execution ProcessID='4' ThreadID='3660'/><Channel>Security</Channel><Computer>windows.default.example.com</Computer><Security/></System><EventData><Data Name='ProcessID'>4</Data><Data Name='Application'>System</Data><Data Name='Direction'>%%14592</Data><Data Name='SourceAddress'>fe80::ffff:ffff:ffff:ffff</Data><Data Name='SourcePort'>0</Data><Data Name='DestAddress'>ff02::1</Data><Data Name='DestPort'>130</Data><Data Name='Protocol'>58</Data><Data Name='FilterRTID'>95023</Data><Data Name='LayerName'>%%14610</Data><Data Name='LayerRTID'>46</Data><Data Name='RemoteUserID'>S-1-0-0</Data><Data Name='RemoteMachineID'>S-1-0-0</Data></EventData><RenderingInfo Culture='en-US'><Message>The Windows Filtering Platform has blocked a connection.\r\n\r\nApplication Information:\r\n\tProcess ID:\t\t4\r\n\tApplication Name:\tSystem\r\n\r\nNetwork Information:\r\n\tDirection:\t\tInbound\r\n\tSource Address:\t\tfe80::ffff:ffff:ffff:ffff\r\n\tSource Port:\t\t0\r\n\tDestination Address:\tff02::1\r\n\tDestination Port:\t\t130\r\n\tProtocol:\t\t58\r\n\r\nFilter Information:\r\n\tFilter Run-Time ID:\t95023\r\n\tLayer Name:\t\tReceive/Accept\r\n\tLayer Run-Time ID:\t46</Message><Level>Information</Level><Task>Filtering Platform Connection</Task><Opcode>Info</Opcode><Channel>Security</Channel><Provider>Microsoft Windows security auditing.</Provider><Keywords><Keyword>Audit Failure</Keyword></Keywords></RenderingInfo></Event>, provider=Microsoft-Windows-Security-Auditing, created=2024-08-07T11:23:05.709Z, kind=event, full_event=The Windows Filtering Platform has blocked a connection.\n\nApplication Information:\n\tProcess ID:\t\t4\n\tApplication Name:\tSystem\n\nNetwork Information:\n\tDirection:\t\tInbound\n\tSource Address:\t\tfe80::ffff:ffff:ffff:ffff\n\tSource Port:\t\t0\n\tDestination Address:\tff02::1\n\tDestination Port:\t\t130\n\tProtocol:\t\t58\n\nFilter Information:\n\tFilter Run-Time ID:\t95023\n\tLayer Name:\t\tReceive/Accept\n\tLayer Run-Time ID:\t46, action=Filtering Platform Connection, outcome=failure}}"
  ],
  "status": 400
}