[RFC] Stage 0: New field set for cyber observables

brett-fitz commented 6 months ago

Overview

This PR begins the RFC process for a new field set for enhancing ECS's ability to handle cyber observables. Stage 0 has been completed as well as a foundational structure for the new field set to aid in the understanding and development.

PR Guidelines

Have you signed the contributor license agreement? Yes
Have you followed the contributor guidelines? Yes
For proposing substantial changes or additions to the schema, have you reviewed the RFC process? Yes
If submitting code/script changes, have you verified all tests pass locally using make test? N/A
If submitting schema/fields updates, have you generated new artifacts by running make and committed those changes? N/A
Is your pull request against main? Unless there is a good reason otherwise, we prefer pull requests against main and will backport as needed. Yes
Have you added an entry to the CHANGELOG.next.md? N/A

Samrose-Ahmed commented 6 months ago

Can you explain how you're defining observable vs indicator, and how this works with multiple observables, perhaps with examples?

brett-fitz commented 6 months ago

@Samrose-Ahmed Thank you for your question! I think it makes sense to define an observable and an indicator similar to how STIX defines them.

Observable

Definition: An observable is the raw data point or entity that can be observed in a network or system.
Examples: IP Addresses, URLs, domain names, file hashes, email addresses, registry keys, process names, etc.
Purpose: Provide raw data of the observable without interpretation or analysis.

Indicator

Definition: An indicator is the combination of the observables and the analysis, providing context and interpretation to potential threats.
Purpose: Guide detection and response efforts. Indicates potential malicious activity based on observed data and associated analysis.

Observable vs Indicator

Observables are purely the data points while indicators are interpretive and actionable. Throughout an investigation, Incident response handlers will collect all the observables seen in an incident from the systems and networks involved. They will then designate some or a composition as indicators and this can change as the investigation continues. Observables provide the foundation, while indicators guide actionable insights. CTI handlers may choose to share observables along side indicators to further understand them and help identify more indicators with their peers.

Examples

Using the STIX2 python library, lets create an observable 192.168.1.1 and also designate it as an indicator.

from stix2 import Indicator, IPv4Address, Relationship, Bundle

# Create an Observable as a Cyber Observable Object
observable_ip = IPv4Address(value="192.168.1.1")

# Create an Indicator
indicator = Indicator(
    name="Malicious activity",
    description="Indicator for a known malicious IP address",
    pattern="[ipv4-addr:value = '192.168.1.1']",
    pattern_type="stix",
    valid_from="2023-01-01T12:00:00Z"
)

# Create a Relationship between the Indicator and the Observable
relationship = Relationship(indicator, 'indicates', observable_ip)

# Create a Bundle to group the objects
bundle = Bundle(objects=[observable_ip, indicator, relationship])

# Display the Bundle in JSON format
print(bundle.serialize(pretty=True))

Output

{
    "type": "bundle",
    "id": "bundle--f6f62144-97f3-4575-a7fb-8ff8be90e51b",
    "objects": [
        {
            "type": "ipv4-addr",
            "spec_version": "2.1",
            "id": "ipv4-addr--cd2ddd9b-6ae2-5d22-aec9-a9940505e5d5",
            "value": "192.168.1.1"
        },
        {
            "type": "indicator",
            "spec_version": "2.1",
            "id": "indicator--ffe366f3-1b95-46b9-b2ec-f099cc677464",
            "created": "2024-01-04T15:43:09.043988Z",
            "modified": "2024-01-04T15:43:09.043988Z",
            "name": "Malicious activity",
            "description": "Indicator for a known malicious IP address",
            "pattern": "[ipv4-addr:value = '192.168.1.1']",
            "pattern_type": "stix",
            "pattern_version": "2.1",
            "valid_from": "2023-01-01T12:00:00Z"
        },
        {
            "type": "relationship",
            "spec_version": "2.1",
            "id": "relationship--94bc0be4-711e-46ba-9cba-974a14a9cd4e",
            "created": "2024-01-04T15:43:09.046889Z",
            "modified": "2024-01-04T15:43:09.046889Z",
            "relationship_type": "indicates",
            "source_ref": "indicator--ffe366f3-1b95-46b9-b2ec-f099cc677464",
            "target_ref": "ipv4-addr--cd2ddd9b-6ae2-5d22-aec9-a9940505e5d5"
        }
    ]
}

Because we created an observable object with the indicator object and built the relationship between the two, this allows us to connect incidents that involve the same indicator using the observable object. The observable object and its ID is deterministic as it is based on the raw data point, the ip address. The indicator ID is randomized because its purpose is to represent point in time contextual information on an observable or composition of observables.

We have conformed this to our observable field set with ECS's threat fields like so:

{
    "observable": {
        "ip": "192.168.1.1",
        "name": "192.168.1.1",
        "provider": ["logs.zeek"],
        "type": "ipv4-addr",
        "spec_version": "2.1",
        "id": "ipv4-addr--cd2ddd9b-6ae2-5d22-aec9-a9940505e5d5"
    },
    "threat": {
        "indicator": {
            "ip": "192.168.1.1",
            "name": "192.168.1.1",
            "type": "ipv4-addr",
            "spec_version": "2.1",
            "id": "indicator--ffe366f3-1b95-46b9-b2ec-f099cc677464",
            "confidence": "High",
            "provider": "OpenCTI",
            "modified_at": "2023-01-17T05:53:42.851Z",
            "marking": {
                "tlp": "AMBER"
            }
        }
    }
}

NOTE: There should only be one observable per log.

Another example to use of a similar implementation is the official threat intel integration from Elastic for OpenCTI. Note, there may be chances where they can simplify their observable field set similar to the threat indicator field set where (e.g. opencti.observable.domain_name.value --> observable.domain)

Please let me know if this answers your questions and if you have any other! 😄

github-actions[bot] commented 3 months ago

This PR is stale because it has been open for 60 days with no activity.

elastic / ecs