Suricata integration with PALANTIR

[ghost]

DESCRIPTION

The objective is to add Suricata as a Source in PALANTIR, the proposed use cases for Suricata are:

• Netflow labeling (Use case 1)

  Description

  The objective is to use the output given by suricata to label Netflowv9 traffic so can be used to constantly train the algorithms for Threat classification.

  Questions

  • Need to create a model for the chosen Suricata output (probably Eve)?
  • How to correlate Netflow and Suricata received information, are they received near in time, will the timestamps match?

• Routing based on Suricata (Use case 2)

  Description

  The objective is to use any present field in Suricata output to decide to what component of the PALANTIR system is going to be sent by writting it into the corresponding topic.

  Questions

  • What field do we need to select for the routing?
  • To what component is going what specific message from Suricata?

NOTES

Antonio Lopez Martinez has provided some examples of data format for Suricata (Eve.json and fast.log): Eve.json:

{
 {
  "timestamp": "2022-05-18T09:16:24.132366+0000",
  "flow_id": 2200967691312447,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "10.1.42.168",
  "src_port": 59854,
  "dest_ip": "91.189.91.38",
  "dest_port": 80,
  "proto": "TCP",
  "tx_id": 1,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2013504,
    "rev": 6,
    "signature": "ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management",
    "category": "Not Suspicious Traffic",
    "severity": 3,
    "metadata": {
      "created_at": [
        "2011_08_31"
      ],
      "former_category": [
        "POLICY"
      ],
      "updated_at": [
        "2020_04_22"
      ]
    }
  },
  "http": {
    "hostname": "archive.ubuntu.com",
    "url": "/ubuntu/pool/universe/j/jq/libjq1_1.6-1ubuntu0.20.04.1_amd64.deb",
    "http_user_agent": "Debian APT-HTTP/1.3 (2.0.6)",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "length": 0
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 50,
    "pkts_toclient": 12,
    "bytes_toserver": 3781,
    "bytes_toclient": 12943,
    "start": "2022-05-18T09:16:23.466239+0000"
  }
}
{
  "timestamp": "2022-05-18T09:16:24.132366+0000",
  "flow_id": 2200967691312447,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "10.1.42.168",
  "src_port": 59854,
  "dest_ip": "91.189.91.38",
  "dest_port": 80,
  "proto": "TCP",
  "tx_id": 2,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2013504,
    "rev": 6,
    "signature": "ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management",
    "category": "Not Suspicious Traffic",
    "severity": 3,
    "metadata": {
      "created_at": [
        "2011_08_31"
      ],
      "former_category": [
        "POLICY"
      ],
      "updated_at": [
        "2020_04_22"
      ]
    }
  },
  "http": {
    "hostname": "archive.ubuntu.com",
    "url": "/ubuntu/pool/universe/j/jq/jq_1.6-1ubuntu0.20.04.1_amd64.deb",
    "http_user_agent": "Debian APT-HTTP/1.3 (2.0.6)",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "length": 0
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 50,
    "pkts_toclient": 12,
    "bytes_toserver": 3781,
    "bytes_toclient": 12943,
    "start": "2022-05-18T09:16:23.466239+0000"
  }
}
{
  "timestamp": "2022-05-18T09:18:22.061737+0000",
  "flow_id": 395037030289208,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "10.1.42.168",
  "src_port": 45686,
  "dest_ip": "52.222.149.33",
  "dest_port": 80,
  "proto": "TCP",
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2013028,
    "rev": 7,
    "signature": "ET POLICY curl User-Agent Outbound",
    "category": "Attempted Information Leak",
    "severity": 2,
    "metadata": {
      "created_at": [
        "2011_06_14"
      ],
      "updated_at": [
        "2022_05_03"
      ]
    }
  },
  "http": {
    "hostname": "testmynids.org",
    "url": "/uid/index.html",
    "http_user_agent": "curl/7.68.0",
    "http_content_type": "text/html",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 39
  },
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 4,
    "pkts_toclient": 3,
    "bytes_toserver": 364,
    "bytes_toclient": 743,
    "start": "2022-05-18T09:18:21.986936+0000"
  }
}
{
  "timestamp": "2022-05-18T09:18:22.098166+0000",
  "flow_id": 395037030289208,
  "in_iface": "eth0",
  "event_type": "alert",
  "src_ip": "52.222.149.33",
  "src_port": 80,
  "dest_ip": "10.1.42.168",
  "dest_port": 45686,
  "proto": "TCP",
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 2100498,
    "rev": 7,
    "signature": "GPL ATTACK_RESPONSE id check returned root",
    "category": "Potentially Bad Traffic",
    "severity": 2,
    "metadata": {
      "created_at": [
        "2010_09_23"
      ],
      "updated_at": [
        "2010_09_23"
      ]
    }
  },
  "http": {
    "hostname": "testmynids.org",
    "url": "/uid/index.html",
    "http_user_agent": "curl/7.68.0",
    "http_content_type": "text/html",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 39
  },
  "files": [
    {
      "filename": "/uid/index.html",
      "sid": [],
      "gaps": false,
      "state": "CLOSED",
      "stored": false,
      "size": 39,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 5,
    "pkts_toclient": 4,
    "bytes_toserver": 430,
    "bytes_toclient": 809,
    "start": "2022-05-18T09:18:21.986936+0000"
  }
}

fast.log:

05/18/2022-09:16:24.132366  [**] [1:2013504:6] ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 10.1.42.168:59854 -> 91.189.91.38:80
05/18/2022-09:16:24.132366  [**] [1:2013504:6] ET POLICY GNU/Linux APT User-Agent Outbound likely related to package management [**] [Classification: Not Suspicious Traffic] [Priority: 3] {TCP} 10.1.42.168:59854 -> 91.189.91.38:80
05/18/2022-09:18:22.061737  [**] [1:2013028:7] ET POLICY curl User-Agent Outbound [**] [Classification: Attempted Information Leak] [Priority: 2] {TCP} 10.1.42.168:45686 -> 52.222.149.33:80
05/18/2022-09:18:22.098166  [**] [1:2100498:7] GPL ATTACK_RESPONSE id check returned root [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 52.222.149.33:80 -> 10.1.42.168:45686

DOCUMENTATION

• Full documentation of the output of Suricata
• Focusing on Eve format the documentation provides the configurations needed to get Eve format as an output
• As our work will mainly focus in the modelling side of the output, the documentation also provides information about the fields in the Eve JSON Format

[idomingu]

Won't do.