search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.11k stars 4.91k forks source link

When using the threatintel module, and misp data the fields do not accept cidr notation #29949

Closed toddferg closed 1 year ago

toddferg commented 2 years ago

Operating System: Centos 8.

Version: Filebeat 7.16.3. and Elasticsearch 7.16.3 Module: Threatintel Sub Config: Misp.

Depending on the data from the Misp source, some of the ip's could be in cidr notation specifically when using the ip address field, it doesn't accept the cidr notation as a valid ip.

example field:

threatintel.indicator.ip

Quick test to check with elasticsearch: 


PUT test
{
  "mappings": {
    "properties": {
      "ip": {
        "type": "ip"
      }
    }
  }
}

POST test/_doc
{
  "ip": "192.168.50.0/24"
}

POST test/_doc
{
  "ip": "192.168.50.2"
}

GET test/_search
{
  "query": {
    "term": {
      "ip": {
        "value": "192.168.50.0/24"
      }
    }
  }
}

DELETE test

In general the misp data might include cidrs notation and the beat will have to work with that information.

elasticmachine commented 2 years ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 commented 2 years ago

https://github.com/elastic/ecs/issues/86 is related.

@toddferg, what behaviour would you like to see here? At the moment I think all we can to is broaden grok pattern tolerance/add disects to allow 192.168.50.0 to be kept, but lose the CIDR prefix size. This would allow this class of search to succeed, but loses the IP range.

Also, do you have example log lines that have this type of input?

toddferg commented 2 years ago

@efd6 Sorry for the delay.

So I'm not sure what would be best. Because siem rules might do best to use the ip range searches which would work?

In regards to log lines, what do you mean? From the misp input, or the output from the beat to elastic?

efd6 commented 2 years ago

So I'm not sure what would be best. Because siem rules might do best to use the ip range searches which would work?

I guess it could be conditionally put into network.cidr though that has not been added yet. Knowing the context of the information coming in would be helpful.

In regards to log lines, what do you mean? From the misp input, or the output from the beat to elastic?

Both? at least whichever you have available.

brsolomon-deloitte commented 1 year ago

We are seeing this as well; for example,

{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [threat.indicator.ip] of type [ip] in document with id 'PaaH0fjVM3nSacGwE2UWYchSYbk='. Preview of field's value: '160.202.208.0/22'\"

The comments here are missing the crux of the issue. MISP is passing a CIDR range, but the Elasticsearch ip type does not accept a /suffix. You can prove this easily in Dev Tools:

PUT my-index-000001
{
  "mappings": {
    "properties": {
      "ip_addr": {
        "type": "ip"
      }
    }
  }
}

PUT my-index-000001/_doc/1
{
  "ip_addr": "103.220.232.0/22"
}

Causes error:

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse field [ip_addr] of type [ip] in document with id '1'. Preview of field's value: '103.220.232.0/22'"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse field [ip_addr] of type [ip] in document with id '1'. Preview of field's value: '103.220.232.0/22'",
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "'103.220.232.0/22' is not an IP string literal."
    }
  },
  "status": 400
}

And to be fair to Elasticseach: this is not an IP address; it is an IP network. It will be correctly rejected by Python's ipaddress module too.

>>> import ipaddress
>>> ipaddress.IPv4Address("103.220.232.0/22")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ipaddress.py", line 1314, in __init__
    raise AddressValueError(f"Unexpected '/' in {address!r}")
ipaddress.AddressValueError: Unexpected '/' in '103.220.232.0/22'
>>> ipaddress.IPv4Network("103.220.232.0/22")
IPv4Network('103.220.232.0/22')

I suggest either:

brsolomon-deloitte commented 1 year ago

This should be classified as a bug in https://github.com/elastic/beats/tree/main/x-pack/filebeat/module/threatintel/misp and the pipeline at https://github.com/elastic/beats/blob/main/x-pack/filebeat/module/threatintel/misp/ingest/pipeline.yml should be fixed to get rid of this error.

brsolomon-deloitte commented 1 year ago

@efd6 It's a major bug because it causes MISP data to be rejected outright and not even ingested.

brsolomon-deloitte commented 1 year ago

I guess it could be conditionally put into network.cidr though that has not been added yet

This is still not part of ECS as of now either.

brsolomon-deloitte commented 1 year ago

Also, do you have example log lines that have this type of input?

Full exception:

{"log.level":"warn","@timestamp":"2023-01-05T14:23:11.585Z","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":429},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.January, 5, 14, 23, 7, 468581449, time.Local), Meta:{\"pipeline\":\"filebeat-8.5.3-threatintel-misp-pipeline\"}, Fields:{\"agent\":{\"ephemeral_id\":\"b195b798-43ff-43f5-a429-204f82cd19f5\",\"id\":\"05906d4e-05e5-406f-9687-552efda1a6d0\",\"name\":\"threatintel8-eck-beats-beat-filebeat-8568dc8d66-4l77z\",\"type\":\"filebeat\",\"version\":\"8.5.3\"},\"ecs\":{\"version\":\"1.12.0\"},\"event\":{\"created\":\"2023-01-05T14:23:07.468Z\",\"dataset\":\"threatintel.misp\",\"module\":\"threatintel\",\"timezone\":\"+00:00\"},\"fileset\":{\"name\":\"misp\"},\"input\":{\"type\":\"httpjson\"},\"message\":\"{\\\"Event\\\":{\\\"Attribute\\\":{\\\"Galaxy\\\":[],\\\"ShadowAttribute\\\":[],\\\"category\\\":\\\"Network activity\\\",\\\"comment\\\":\\\"\\\",\\\"deleted\\\":false,\\\"disable_correlation\\\":false,\\\"distribution\\\":\\\"5\\\",\\\"event_id\\\":\\\"1528\\\",\\\"first_seen\\\":null,\\\"id\\\":\\\"4081012\\\",\\\"last_seen\\\":null,\\\"object_id\\\":\\\"0\\\",\\\"object_relation\\\":null,\\\"sharing_group_id\\\":\\\"0\\\",\\\"timestamp\\\":\\\"1670293864\\\",\\\"to_ids\\\":true,\\\"type\\\":\\\"ip-dst\\\",\\\"uuid\\\":\\\"4fa3610f-5412-41dd-9034-c8294fc3c7c2\\\",\\\"value\\\":\\\"146.88.240.0/24\\\"},\\\"CryptographicKey\\\":[],\\\"EventReport\\\":[],\\\"Galaxy\\\":[],\\\"Object\\\":[],\\\"Org\\\":{\\\"id\\\":\\\"1\\\",\\\"local\\\":true,\\\"name\\\":\\\"ORGNAME\\\",\\\"uuid\\\":\\\"121f5d02-2e2e-4180-9166-fcad7e01a20d\\\"},\\\"Orgc\\\":{\\\"id\\\":\\\"1\\\",\\\"local\\\":true,\\\"name\\\":\\\"ORGNAME\\\",\\\"uuid\\\":\\\"121f5d02-2e2e-4180-9166-fcad7e01a20d\\\"},\\\"RelatedEvent\\\":[{\\\"Event\\\":{\\\"Org\\\":{\\\"id\\\":\\\"1\\\",\\\"name\\\":\\\"ORGNAME\\\",\\\"uuid\\\":\\\"121f5d02-2e2e-4180-9166-fcad7e01a20d\\\"},\\\"Orgc\\\":{\\\"id\\\":\\\"3\\\",\\\"name\\\":\\\"CIRCL\\\",\\\"uuid\\\":\\\"55f6ea5e-2c60-40e5-964f-47a8950d210f\\\"},\\\"analysis\\\":\\\"2\\\",\\\"date\\\":\\\"2018-03-26\\\",\\\"distribution\\\":\\\"3\\\",\\\"id\\\":\\\"896\\\",\\\"info\\\":\\\"OSINT - Forgot About Default Accounts? No Worries, GoScanSSH Didn’t\\\",\\\"org_id\\\":\\\"1\\\",\\\"orgc_id\\\":\\\"3\\\",\\\"published\\\":true,\\\"threat_level_id\\\":\\\"3\\\",\\\"timestamp\\\":\\\"1523865236\\\",\\\"uuid\\\":\\\"5acdb4d0-b534-4713-9612-4a1d950d210f\\\"}}],\\\"ShadowAttribute\\\":[],\\\"Tag\\\":[{\\\"colour\\\":\\\"#004577\\\",\\\"exportable\\\":true,\\\"hide_tag\\\":false,\\\"id\\\":\\\"1\\\",\\\"is_custom_galaxy\\\":false,\\\"is_galaxy\\\":false,\\\"local\\\":0,\\\"local_only\\\":false,\\\"name\\\":\\\"osint:source-type=\\\\\\\"block-or-filter-list\\\\\\\"\\\",\\\"numerical_value\\\":null,\\\"user_id\\\":\\\"0\\\"}],\\\"analysis\\\":\\\"2\\\",\\\"attribute_count\\\":\\\"2053\\\",\\\"date\\\":\\\"2022-07-26\\\",\\\"disable_correlation\\\":false,\\\"distribution\\\":\\\"0\\\",\\\"event_creator_email\\\":\\\"admin@admin.test\\\",\\\"extends_uuid\\\":\\\"\\\",\\\"id\\\":\\\"1528\\\",\\\"info\\\":\\\"firehol_level1 feed\\\",\\\"locked\\\":false,\\\"org_id\\\":\\\"1\\\",\\\"orgc_id\\\":\\\"1\\\",\\\"proposal_email_lock\\\":false,\\\"protected\\\":null,\\\"publish_timestamp\\\":\\\"0\\\",\\\"published\\\":false,\\\"sharing_group_id\\\":\\\"0\\\",\\\"threat_level_id\\\":\\\"4\\\",\\\"timestamp\\\":\\\"1672885819\\\",\\\"uuid\\\":\\\"2ed041a1-33ea-4e54-bf26-bf1c7ce191b3\\\"}}\",\"service\":{\"type\":\"threatintel\"},\"tags\":[\"threatintel-misp\",\"forwarded\"]}, Private:(*cursor.updateOp)(0xc0242b8cc0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [threat.indicator.ip] of type [ip] in document with id '2ZLj0k/Z4pZz7KHKJz47cOXESnA='. Preview of field's value: '146.88.240.0/24'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"'146.88.240.0/24' is not an IP string literal.\"}}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}
brsolomon-deloitte commented 1 year ago

Here's the record prettified with jq

{
  "Event": {
    "Attribute": {
      "Galaxy": [],
      "ShadowAttribute": [],
      "category": "Network activity",
      "comment": "",
      "deleted": false,
      "disable_correlation": false,
      "distribution": "5",
      "event_id": "1528",
      "first_seen": null,
      "id": "4081012",
      "last_seen": null,
      "object_id": "0",
      "object_relation": null,
      "sharing_group_id": "0",
      "timestamp": "1670293864",
      "to_ids": true,
      "type": "ip-dst",
      "uuid": "4fa3610f-5412-41dd-9034-c8294fc3c7c2",
      "value": "146.88.240.0/24"
    },
    "CryptographicKey": [],
    "EventReport": [],
    "Galaxy": [],
    "Object": [],
    "Org": {
      "id": "1",
      "local": true,
      "name": "ORGNAME",
      "uuid": "121f5d02-2e2e-4180-9166-fcad7e01a20d"
    },
    "Orgc": {
      "id": "1",
      "local": true,
      "name": "ORGNAME",
      "uuid": "121f5d02-2e2e-4180-9166-fcad7e01a20d"
    },
    "RelatedEvent": [
      {
        "Event": {
          "Org": {
            "id": "1",
            "name": "ORGNAME",
            "uuid": "121f5d02-2e2e-4180-9166-fcad7e01a20d"
          },
          "Orgc": {
            "id": "3",
            "name": "CIRCL",
            "uuid": "55f6ea5e-2c60-40e5-964f-47a8950d210f"
          },
          "analysis": "2",
          "date": "2018-03-26",
          "distribution": "3",
          "id": "896",
          "info": "OSINT - Forgot About Default Accounts? No Worries, GoScanSSH Didn’t",
          "org_id": "1",
          "orgc_id": "3",
          "published": true,
          "threat_level_id": "3",
          "timestamp": "1523865236",
          "uuid": "5acdb4d0-b534-4713-9612-4a1d950d210f"
        }
      }
    ],
    "ShadowAttribute": [],
    "Tag": [
      {
        "colour": "#004577",
        "exportable": true,
        "hide_tag": false,
        "id": "1",
        "is_custom_galaxy": false,
        "is_galaxy": false,
        "local": 0,
        "local_only": false,
        "name": "osint:source-type=\"block-or-filter-list\"",
        "numerical_value": null,
        "user_id": "0"
      }
    ],
    "analysis": "2",
    "attribute_count": "2053",
    "date": "2022-07-26",
    "disable_correlation": false,
    "distribution": "0",
    "event_creator_email": "admin@admin.test",
    "extends_uuid": "",
    "id": "1528",
    "info": "firehol_level1 feed",
    "locked": false,
    "org_id": "1",
    "orgc_id": "1",
    "proposal_email_lock": false,
    "protected": null,
    "publish_timestamp": "0",
    "published": false,
    "sharing_group_id": "0",
    "threat_level_id": "4",
    "timestamp": "1672885819",
    "uuid": "2ed041a1-33ea-4e54-bf26-bf1c7ce191b3"
  }
}