Splunk KVStore not so STIX-friendly because UUID in fields name prevent to extract sub-fields with a Splunk Lookup Definition

adel-akloul-mox commented 1 year ago

Use case

Given the below STIX sent to a Splunk KVSTORE, note the presence of an UUID to fields such as extensions--UUID

  {
    "id": "domain-name--b9a6795a-3118-5fb3-a01a-2d360d20c432",
    "spec_version": "2.1",
    "type": "domain-name",
    "extensions": {
      "extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba": {
        "extension_type": "property-extension",
        "id": "01a0321c-a8a3-4092-b404-27187fd1c440",
        "type": "Domain-Name",
        "created_at": "2023-05-15T14:06:33.344Z",
        "updated_at": "2023-06-15T10:33:53.736Z",
        "is_inferred": false,
        "creator_ids": [
          "88ec0c6a-13ce-5e39-b486-354fe4a7084f"
        ]
      },
      "extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82": {
        "extension_type": "property-extension",
        "labels": [
          "vidar",
          "link-manager"
        ],
        "score": 50
      }
    },
    "object_marking_refs": [
      "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"
    ],
    "value": "git.wordpress.msi-afteburner.com",
    "stream_name": "live",
    "_user": "nobody",
    "_key": "01a0321c-a8a3-4092-b404-27187fd1c440"
  },

Issues:

The First issue is Splunk KVStore not supporting WILDCARD for field name extraction (it is only supported for field value); As a result fields such as extensions.extension-definition--*.[labels, score, created_at, updated_at] cannot be extracted via a splunk Lookup Definition
Second issue relates to Splunk licensing model (based on daily volume ingested in the platform) and as a result it will be more license effective to reduce the size of the payload by stripping away unecessary fields

Current Workaround

A workaround for Lookup Definition not being able to extract fields having an uuid, is to use instead Splunk Search Processing Language (SPL) to extract fields, and maybe send it to another KVStore
A workaround for Daily ingestion volume in Splunk is to remove unecessary STIX with SPLUNK_IGNORE_TYPES

Proposed Solution

My proposed solution is to amend connector-splunk so that it changes the payload format in a more KVstore-friendly manner as it streams it to Splunk. The proposed payload will looks like:

  {
    "id": "domain-name--b9a6795a-3118-5fb3-a01a-2d360d20c432",
    "type": "domain-name",
    "created_at": "2023-05-15T14:06:33.344Z",
    "updated_at": "2023-06-15T10:33:53.736Z",
    "labels": [
      "vidar",
      "link-manager"
    ],
    "score": 50
    "value": "evil-domain.com",
    "stream_name": "live",
    "_user": "nobody",
    "_key": "01a0321c-a8a3-4092-b404-27187fd1c440"
  },

On top of that, I propose to modify the SPLUNK_IGNORE_TYPES environment variable since apart domains, ipv4, ipv6, url and hashes, all other STIX objects are not so relevant for threat hunting:

SPLUNK_IGNORE_TYPES: label,marking-definition,identity,vulnerability,external-reference,vocabulary,threat-actor,relationship,indicator,attack-pattern,malware,intrusion-set

Benefits of such a changes:

drastically reduced volume sent to Splunk; hence splunk daily ingestion license saving
ALL fields can easily be extracted by Splunk Lookup Definition
No need for extra-processing with SPL or second kvstore to extract fields within extensions--UUID
A splunk engineers can leverage:
- The new friendly format to append openCTI IOC to ThreatGen kvstore to be used in combination with Splunk ES (Enterprise Security)
- Date fields to implement eviction mechanism (such as IOC older then 6 months)
- Labels and Score to implement specific actions with SOAR, or IOC eviction ... etc ...

Additional Information

Similar transformation would be made for type:

ipv4-addr
url
domain-name

  {
    "id": "ipv4-addr--01863713-15de-50d7-9c8a-3c32c4294966",
    "type": "ipv4-addr",
    "created_at": "2023-05-16T01:37:23.057Z",
    "updated_at": "2023-06-15T10:33:35.826Z",
    "labels": [
      "mirai"
    ],
    "score": 50
    "value": "201.182.144.41",
    "stream_name": "live",
    "_user": "nobody",
    "_key": "01b58111-0021-42aa-916d-aae693ccac31"
  },
  {
    "id": "url--406cfabb-344a-5c55-98c9-9e266bfc0d3a",
    "type": "url",
    "created_at": "2023-05-16T08:01:50.069Z",
    "updated_at": "2023-06-15T10:34:11.446Z",
    "labels": [
          "hygiene",
    ],
    "score": 50
    "value": "http://o6ia6w.cloudphonoway.online/",
    "score": 50,
    "stream_name": "live",
    "_user": "nobody",
    "_key": "01da9e18-bdd9-408f-b2cc-44f8ececdfbd"
  },

For type: "file", I propose further changes to normalize the format by replacing the type with the actual hash-type:

  {
    "id": "file--03efa8d9-a0b2-535e-842a-7b0aa4278e38",
    "type": "SHA-256",
    "created_at": "2023-05-15T17:10:25.169Z",
    "updated_at": "2023-06-15T10:33:26.489Z",
    "score": 50
    "value": "d087031f8949648ce043658b8f3feb016f7c082feab001346bf12b1cddb91397"
    "stream_name": "live",
    "_user": "nobody",
    "_key": "01eda4a4-52f6-456e-a012-a6e76489f346"
  },

instead of

  {
    "id": "file--03efa8d9-a0b2-535e-842a-7b0aa4278e38",
    "spec_version": "2.1",
    "type": "file",
    "extensions": {
      "extension-definition--ea279b3e-5c71-4632-ac08-831c66a786ba": {
        "extension_type": "property-extension",
        "id": "01eda4a4-52f6-456e-a012-a6e76489f346",
        "type": "StixFile",
        "created_at": "2023-05-15T17:10:25.169Z",
        "updated_at": "2023-06-15T10:33:26.489Z",
        "is_inferred": false,
        "creator_ids": [
          "88ec0c6a-13ce-5e39-b486-354fe4a7084f"
        ]
      },
      "extension-definition--f93e2c80-4231-4f9a-af8b-95c9bd566a82": {
        "extension_type": "property-extension",
        "score": 50
      }
    },
    "object_marking_refs": [
      "marking-definition--613f2e26-407d-48c7-9eca-b8e91df99dc9"
    ],
    "hashes": {
      "SHA-256": "d087031f8949648ce043658b8f3feb016f7c082feab001346bf12b1cddb91397"
    },
    "stream_name": "live",
    "_user": "nobody",
    "_key": "01eda4a4-52f6-456e-a012-a6e76489f346"
  },

Would you be willing to submit a PR?

I am happy to submit a PR if the described change is accepted by the openCTI-Splunk community since it will considerably improve our Security Operations and SOAR automation.
I am happy to be looped in with other connector-splunk user and dev-team to further groom the proposed solution.

adel-akloul-mox commented 1 year ago

@SamuelHassine happy to hear how you receive the idea before I implement the change

SamuelHassine commented 1 year ago

Hello @adel-akloul-mox,

Fully agree with all the proposed changes.

Kind regards, Samuel

aakloul commented 1 year ago

PR for your review: https://github.com/OpenCTI-Platform/connectors/pull/1407

OpenCTI-Platform / connectors