Create ECS compliant index templates

AlexRuiz7 commented 1 week ago

Description

As part of the Data Persistence Model Redesign, new indices will appear in the Wazuh Indexer. We want these indices to be created automatically after Wazuh Indexer boots up. For that, we'll build a plugin.

As owners of the Wazuh Indexer, we will define the indices settings and fields. These need to be ECS compliant.

Tasks

[x] Define the index data model (template) for the agents index.
[x] Define the index data model (template) for the commands index.
[ ] Define the index data model (template) for the wazuh-alerts (stateless) index.
[ ] Define the index data model (template) for the wazuh-monitoring (stateless) index.
[ ] Define the index data model (template) for the wazuh-statistics (stateless) index.
[ ] https://github.com/wazuh/wazuh-indexer/issues/282
[ ] Define the index data model (template) for the stateful indices.
- [x] Vulnerability
- [x] FIM
- [x] Inventory
- [ ] SCA

AlexRuiz7 commented 1 week ago

`agents` index data model

Fields summary

The fields are based on https://github.com/wazuh/wazuh/issues/23396#issuecomment-2176402993

Based on ECS Agent Fields.

Field	ECS field	Type	Description
uuid	`agent.id`	keyword	Agent's ID
name	`agent.name`	keyword	Agent's name
groups	*`agent.groups`	keyword	Agent's groups
internal_key	*`agent.key`	keyword	Agent's registration key
type	`agent.type`	keyword	??
version	`agent.version`	keyword	Agent's version
connection_status	*`agent.last_login`	date	Agent's last login
node_name	*`agent.persistent_connection_node`	keyword	The Wazuh Server node with which the Agent has a persistent connection

* Custom field

ECS mapping

---
- name: agent
  fields:
    agent:
        fields:
            id: {}
            name: {}
            type: {}
            version: {}

---
- name: agent
  title: Wazuh Agents
  short: Wazuh Inc. custom fields.
  type: group
  fields:
    - name: groups
      type: keyword
      description: >
          The groups the agent belongs to.
    - name: key
        type: keyword
        description: >
            The agent's registration key.
    - name: last_login
        type: date
        description: >
            The agent's last login.
    - name: persistent_connection_node
        type: keyword
        description: >
            The Wazuh Server node with which the agent has a persistent connection.

Index settings

{
    "index_patterns": ["agents"],
    "priority": 1,
    "template": {
        "settings": {
            "index": {
                "codec": "zstd",
                "number_of_shards": "1",
                "number_of_replicas": "0",
                "refresh_interval": "5s",
                "query.default_field": [
                    "agent.id",
                    "agent.name",
                    "agent.type",
                    "agent.version",
                    "agent.name"
                ]
            }
        }
    }
}

AlexRuiz7 commented 1 week ago

`wazuh-alerts-5.x` time series index

[!NOTE] Pending review. Fields may change.

Stateless index.

Fields summary

The fields are based on https://github.com/wazuh/wazuh/blob/11334-dev-new-wazuh-engine/src/engine/extension/elasticsearch/7.x/wazuh-template.json

Most of the ECS fields are in use. For a complete list, check the next section.

There is also a custom category wazuh with the following fields:

queue
location
message
origin
resgistered_ip
decoders
source
syscollector
rootcheck
isInternal

* We have to review these fields.

ECS mapping

---
name: stateless
fields:
  base: "*"
  agent: "*"
  client: "*"
  cloud: "*"
  container: "*"
  data_stream: "*"
  destination: "*"
  device: "*"
  dll: "*"
  dns: "*"
  ecs: "*"
  email: "*"
  error: "*"
  event: "*"
  faas: "*"
  file: "*"
  group: "*"
  host: "*"
  http: "*"
  log: "*"
  network: "*"
  observer: "*"
  organization: "*"
  package: "*"
  process: "*"
  registry: "*"
  related: "*"
  rule: "*"
  server: "*"
  service: "*"
  source: "*"
  threat: "*"
  tls: "*"
  tracing: "*"
  url: "*"
  user: "*"
  user_agent: "*"
  vulnerability: "*"
  wazuh (custom): "*"

{
  "index_patterns": [
    "wazuh-alerts-5.x-*"
  ],
  "priority": 1,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": 2000
          }
        },
        "refresh_interval": "5s",
    }
  }
}

AlexRuiz7 commented 1 week ago

The index templates above are in draft. When reviewed and approved. I'll create the tooling to generate these index templates automatically.

See https://github.com/wazuh/wazuh-indexer/blob/4.9.0/ecs

AlexRuiz7 commented 3 days ago

`commands` index data model

Fields summary

This index stores information about the commands executed by the agents. The index appears in 5.0.0 for the first time.

Based on ECS Process Fields.

ECS field	Type	Description
*`command.args`	match_only_text	Array of process arguments, starting with the absolute path to the executable.
*`command.agent.id`	keyword	Agent’s ID to send the command to.
*`command.status`	keyword	One of: ["pending", "sent", "completed", "failed"]**
*`command.info`	match_only_text	additional information of the execution of the command. For example, the error message.

* Custom field.

** The status of the command execution:

Pending: The command is waiting to be sent to the agent.
Sent: The command was sent to the agent.
Completed: The command was executed successfully.
Failed: The command execution failed.

ECS mapping

---
- name: commands
  title: Commands
  description: >
    This index stores information about the commands executed by the agents.
  fields:
    - name: args
      type: match_only_text
      level: custom
      description: >
        Array of process arguments, starting with the absolute path to the executable.
        Example: ["/usr/bin/ssh", "-l", "user", "10.0.0.16"]
    - name: agent.id
      type: keyword
      level: custom
      description: >
        Agent’s ID to send the command to.
    - name: status
        type: keyword
        level: custom
        description: >
            The status of the command execution.
            Possible values: ["Pending", "Sent", "Completed", "Failed"]
    - name: info
        type: match_only_text
        level: custom
        description: >
            Additional information of the execution of the command. For example, the error message.

Index settings

{
    "index_patterns": ["commands-*"],
    "priority": 1,
    "template": {
        "settings": {
            "index": {
                "codec": "zstd",
                "number_of_shards": "1",
                "number_of_replicas": "0",
                "refresh_interval": "5s",
                "query.default_field": [
                    "args",
                    "agent.id",
                    "status",
                    "info"
                ]
            }
        }
    }
}

AlexRuiz7 commented 3 days ago

`stateful-fim` index data model

Fields summary

The fields are based on https://github.com/wazuh/wazuh-indexer/issues/282#issuecomment-2189377542

Based on ECS:

Field	ECS field	Type	Description
arch	* ?	keyword	Is arch a file property?
attributes	`file.attributes`	keyword	Array of file attributes.
file	`file.name`	keyword	Name of the file including the extension, without the directory.
full_path	`file.path`	keyword	Full path to the file, including the file name.
gid	`file.gid`	keyword	Primary group ID (GID) of the file.
gname	`file.group`	keyword	Primary group name of the file.
inode	`file.inode`	keyword	Inode representing the file in the filesystem.
md5	`file.hash.md5`	keyword	MD5 hash of the file.
mtime	`file.mtime`	date	Last time the file's metadata changed.
perm	`file.mode`	keyword	File permissions in octal mode.
sha1	`file.hash.sha1`	keyword	SHA1 hash of the file.
sha256	`file.hash.sha256`	keyword	SHA256 hash of the file.
size	`file.size`	long	File size in bytes.
symbolic_path	`file.target_path`	keyword	Target path for symlinks.
type	`file.type`	keyword	File type (file, dir, or symlink).
uid	`file.uid`	keyword	User ID (UID) of the file owner.
uname	`file.owner`	keyword	File owner’s username.
value_name	`registry.key`	keyword	Hive-relative path of keys.
value_type	`registry.value`	keyword	Name of the value written.

* Custom field

ECS mapping

---
- name: fim
    fields:
        file:
            fields:
                attributes: {}
                name: {}
                path: {}
                gid: {}
                group: {}
                inode: {}
                hash:
                    fields:
                        md5: {}
                        sha1: {}
                        sha256: {}
                mtime: {}
                mode: {}
                size: {}
                target_path: {}
                type: {}
                uid: {}
                owner: {}
        registry:
            fields:
                key: {}
                value: {}

Index settings

{
    "index_patterns": ["wazuh-states-fim-*"],
    "priority": 1,
    "template": {
        "settings": {
            "index": {
                "codec": "zstd",
                "number_of_shards": "1",
                "number_of_replicas": "0",
                "refresh_interval": "5s",
                "query.default_field": [
                    "file.name",
                    "file.path",
                    "file.target_path",
                    "file.group",
                    "file.uid",
                    "file.gid"
                ]
            }
        }
    }
}

AlexRuiz7 commented 3 days ago

`stateful-vulnerability` index data model

Fields summary

The fields are based on https://github.com/wazuh/wazuh-indexer/blob/4.9.0/ecs/vulnerability-detector

Based on ECS:

ECS field	Type	Description
`agent.build.original`	keyword	Extended build information for the agent.
`agent.ephemeral_id`	keyword	Ephemeral identifier of this agent (if one exists).
`agent.id`	keyword	Unique identifier of this agent (if one exists).
`agent.name`	keyword	Custom name of the agent.
`agent.type`	keyword	Type of the agent.
`agent.version`	keyword	Version of the agent.
`host.os.family`	keyword	OS family (such as redhat, debian, freebsd, windows).
`host.os.full`	keyword	Operating system name, including the version or code name.
`host.os.kernel`	keyword	Operating system kernel version as a raw string.
`host.os.name`	keyword	Operating system name, without the version.
`host.os.platform`	keyword	Operating system platform (such centos, ubuntu, windows).
`host.os.type`	keyword	Use the os.type field to categorize the operating system into one of the broad commercial families.
`host.os.version`	keyword	Operating system version as a raw string.
`message`	match_only_text	A human-readable summary of the event.
`package.architecture`	keyword	Package architecture.
`package.build_version`	keyword	Additional information about the build version of the installed package.
`package.checksum`	keyword	Checksum of the installed package for verification.
`package.description`	keyword	Description of the package.
`package.install_scope`	keyword	Indicating how the package was installed, e.g. user-local, global.
`package.installed`	date	Time when package was installed.
`package.license`	keyword	License under which the package was released.
`package.name`	keyword	Package name
`package.path`	keyword	Path where the package is installed.
`package.reference`	keyword	Home page or reference URL of the software in this package, if available.
`package.size`	long	Package size in bytes.
`package.type`	keyword	Type of package.
`package.version`	keyword	Package version
`tags`	keyword	List of keywords used to tag each event.
*`wazuh.cluster.name`	keyword	Name of the Wazuh cluster.
*`wazuh.cluster.node`	keyword	Name of the Wazuh cluster node.
*`wazuh.manager.name`	keyword	Name of the Wazuh manager.
*`wazuh.schema.version`	keyword	Version of the Wazuh schema.

* Custom field

ECS mapping

---
- name: vulnerability_detector
  fields:
    base:
        fields: 
        tags: []
        message: ""
    agent:
        fields: "*"
    package:
        fields: "*"
    host:
        fields:
            os:
                fields: "*"
    vulnerability:
        fields: "*"
    wazuh:
        fields: "*"

---
- name: vulnerability
  title: Vulnerability
  short: Wazuh Inc. custom fields.
  type: group
  fields:
    - name: detected_at
      type: date
      level: custom
      description: Vulnerability's detection date.
    - name: published_at
      type: date
      level: custom
      description: Vulnerability's publication date.

---
- name: wazuh
  title: Wazuh
  description: >
    Wazuh Inc. custom fields
  fields:
    - name: cluster.name
      type: keyword
      level: custom
      description: Wazuh cluster name.
    - name: cluster.node
      type: keyword
      level: custom
      description: Wazuh cluster node name.
    - name: manager.name
      type: keyword
      level: custom
      description: Wazuh manager name. Used by dashboards to filter results on single node deployments.
    - name: schema.version
      type: keyword
      level: custom
      description: Wazuh schema version.

Index settings

{
    "index_patterns": ["wazuh-states-vulnerabilities-*"],
    "priority": 1,
    "template": {
        "settings": {
            "index": {
                "codec": "zstd",
                "number_of_shards": "1",
                "number_of_replicas": "0",
                "refresh_interval": "5s",
                "query.default_field": [
                    "base.tags",
                    "agent.id",
                    "host.os.family",
                    "host.os.full.text",
                    "host.os.version",
                    "package.name",
                    "package.version",
                    "vulnerability.id",
                    "vulnerability.description.text",
                    "vulnerability.severity",
                    "wazuh.cluster.name"
                ]
            }
        }
    }
}

AlexRuiz7 commented 2 days ago

`stateful-inventory` index data model

Fields summary

The fields are based on https://github.com/wazuh/wazuh-indexer/issues/282#issuecomment-2189837612

Based on ECS:

Field name	ECS field name	Data type	Description
architecture	`host.architecture`	keyword	Operating system architecture.
checksum	`process.hash.md5`	keyword	Scan MD5 hash.
hostname	`host.hostname`	keyword	Hostname of the host.
os_build	`host.os.kernel`	keyword	Operating system kernel version as a raw string.
os_codename	`host.os.full`	keyword	Operating system name, including the version or code name.
os_name	`host.os.name`	keyword	Operating system name, without the version.
os_platform	`host.os.platform`	keyword	Operating system platform (such centos, ubuntu, windows).
os_version	`host.os.version`	keyword	Operating system version as a raw string.
scan_time	`@timestamp`	date	Date/time when the event originated.
sysname	`host.name`	keyword	Name of the host (lowercase FQDN).

Details

Removed fields: - os_display_version - os_major (can be extracted from os_version) - os_minor (can be extracted from os_version) - os_patch (can be extracted from os_version) - os_release - reference - release - scan_id - sysname - version Available fields: - `os_family` - `os.type`

ECS mapping

---
- name: inventory
  fields:
    base:
        fields:
            "@timestamp": {}
    host:
        fields:
            architecture: {}
            hostname: {}
            name: {}
            os:
                fields:
                    kernel: {}
                    full: {}
                    name: {}
                    platform: {}
                    version: {}
    process:
        fields:
            hash:
                fields:
                    md5: {}

Index settings

{
    "index_patterns": ["inventory-*"],
    "priority": 1,
    "template": {
        "settings": {
            "index": {
                "codec": "zstd",
                "number_of_shards": "1",
                "number_of_replicas": "0",
                "refresh_interval": "5s",
                "query.default_field": [
                    "host.name",
                    "host.os.name",
                    "host.os.version",
                ]
            }
        }
    }
}

wazuh / wazuh-indexer