Open sdvendramini opened 4 weeks ago
LucioDonda
commented
3 weeks ago Hi @sdvendramini While I'm looking for them: Have you detected which fields or in which situation did the agent generate any non-ECS compliant event field? Where they part of any particular module ? TIA
GGP1
commented
3 weeks ago @LucioDonda the ECS templates have been modified recently, it is highly likely that the agent is generating events with an outdated format.
I've been working on Update stateful events data models #26568 which covers the same case but for the Communications API POST /events/stateful endpoint.
Here are some of the structures we are accepting in JSON format.
At the same time, those objects have to be inside the data field of a wrapper object that also includes a module field. For example, an inventory package event would look like this:
{
"data": {
"agent": {
"id": "string",
"groups": []
},
"scan_time": "2024-10-28T18:26:10.634Z",
"package": {
"architecture": "string",
"description": "string",
"installed": "2024-10-28T18:26:10.634Z",
"name": "string",
"path": "string",
"size": 0,
"type": "string",
"version": "string"
}
},
"module": "inventory_package"
}If you have any doubts or comments, we can arrange a meeting to discuss this further.
vikman90
commented
3 weeks ago {
"module": {
"name": "inventory",
"type": "package"
},
"data": { ... }
}
{
"module": {
"name": "vulnerability"
},
"data": { ... }
}
{
"module": {
"name": "data"
},
"data": { ... }
}
{
"module": { "name": "logcollector" },
"data": {
"file": { "path": "/var/log/syslog" },
"event": { "original": "2024-10-31T16:21:25.198579+01:00 Rocket systemd-resolved[176]: Clock change detected. Flushing caches." }
}
}
cborla
commented
3 weeks ago The following analysis is based on the following sources.
@dataclass
class OS:
"""OS data model."""
kernel: str
full: str
name: str
platform: str
version: str
type: str
family: str
@dataclass
class Host:
"""Host data model."""
architecture: str
hostname: str
os: OS
@dataclass
class ProcessHash:
md5: str
@dataclass
class Process:
"""Process data model."""
hash: ProcessHash
@dataclass
class InventoryEvent(BaseModel):
"""Inventory events data model."""
host: Host
process: Process
def get_index_name(self) -> str:
"""Get the index name for the event type.
Returns
-------
str
Index name.
"""
return INVENTORY_INDEX
From the above classes of the Inventory Stateful event, we can obtain the following diagram.
InventoryEvent
│
├── Host
│ ├── architecture : str
│ ├── hostname : str
│ └── os : OS
│ ├── kernel : str
│ ├── full : str
│ ├── name : str
│ ├── platform : str
│ ├── version : str
│ ├── type : str
│ └── family : str
│
└── Process
└── hash : ProcessHash
└── md5 : str
There is a very big difference in the amount of data and data structures being shared. Currently the inventory gets 9 types of structures, with their corresponding information.
As a first development, it can be adapted to the structure model proposed by the indexer.
cborla
commented
3 weeks ago cborla
commented
3 weeks ago {
"module": {
"name": "inventory",
"type": "package"
},
"data": { ... }
}
{
"module": {
"name": "vulnerability"
},
"data": { ... }
}Update of the format of the messages to be sent to the server from logcollector.
{
"module": { "name": "logcollector" },
"data": {
"file": { "path": "/var/log/syslog" },
"event": { "original": "2024-10-31T16:21:25.198579+01:00 Rocket systemd-resolved[176]: Clock change detected. Flushing caches." }
}
}Updating of inventory tables and normalisation of fields. (WIP)
cborla
commented
3 weeks ago The agent is currently sending inventory messages in the following format. It remains to adapt the fields according to ECS, but within the format the operation to be carried out must be included.
{
"data":
{
"argvs": null,
"checksum": "ab94278230d240b66082ba6cbf52106cebff41ac",
"cmd": null,
"egroup": "root",
"euser": "root",
"fgroup": "root",
"name": "kworker/u9:0-tt",
"nice": -20,
"nlwp": 1,
"pgrp": 0,
"pid": "86",
"ppid": 2,
"priority": 0,
"processor": 2,
"resident": 0,
"rgroup": "root",
"ruser": "root",
"scan_time": "2024/11/02 01:55:47",
"session": 0,
"sgroup": "root",
"share": 0,
"size": 0,
"start_time": 1730351047,
"state": "I",
"stime": 0,
"suser": "root",
"tgid": 86,
"tty": 0,
"utime": 0,
"vm_size": 0
},
"operation": "DELETED",
"type": "dbsync_processes"
}{
"agent": {
"uuid": "UUID",
"groups": [ ],
"os": "Amazon Linux 2",
"platform": "Linux",
"type": "Endpoint",
"version": "5.0.0",
"ip": "192.168.1.2"
}
}
{
"module": "logcollector",
"type": "file"
}
{
"log": {
"file": {
"path": "string"
}
},
"base": {
"tags": "string"
},
"event": {
"original": "string",
"ingested": "string",
"module": "string",
"provider": "string"
}
}
{
"module": "inventory",
"type": "package"
}
{
"log": {
"file": {
"path": "string"
}
},
"base": {
"tags": "string"
},
"event": {
"original": "string",
"ingested": "string",
"module": "string",
"provider": "string"
}
}
{
"agent": {
"uuid": "UUID",
"groups": [ ],
"os": "Amazon Linux 2",
"platform": "Linux",
"type": "Endpoint",
"version": "5.0.0",
"ip": "192.168.1.2" }
}
{
"module": "inventory",
"type": "package",
"operation": "modified",
"id": "lskdjf023984902358"
}
{
"scan_time": "2024-10-28T18:26:10.634Z",
"package": {
"architecture": "string",
"description": "string",
"installed": "2024-10-28T18:26:10.634Z",
"name": "string",
"path": "string",
"size": 0,
"type": "string",
"version": "string"
}
}
{
"module": "inventory",
"type": "network",
"operation": "add",
"id": "lskdjf023984902358"
}
{
"scan_time": "2024-10-28T18:26:10.634Z",
"package": {
"architecture": "string",
"description": "string",
"installed": "2024-10-28T18:26:10.634Z",
"name": "string",
"path": "string",
"size": 0,
"type": "string",
"version": "string"
}
}
{
"module": "inventory",
"type": "network",
"operation": "delete",
"id": "asdfsdfkdsj98237498325"
}A new column is added to the queue, to store the module metadata.
| New queue structure: | module_name | module_type | metadata | data |
|---|
This will allow the module data to be included in the new object, and will allow the data pair to be shared on a per-event basis.
{
"module": "logcollector",
"type": "file"
}and
{
"module": "inventory",
"type": "network",
"operation": "delete",
"id": "asdfsdfkdsj98237498325"
}
Parent Issue: https://github.com/wazuh/wazuh-agent/issues/241
Description
The events generated by the agent must adhere to the common schema for consistency and compatibility across systems.
Details
Format body stateless and stateful
Stateless
```json { "agent": { "id": "2887e1cf-9bf2-431a-b066-a46860080f56", "name": "agent1", "type": "endpoint", "version": "5.0.0", "groups": ["group1", "group2"], "host": { "hostname": "myhost", "os": { "name": "Amazon Linux 2" "platform": "Linux" }, "ip": ["192.168.1.2"], "architecture": "x86_64" } } } { "module": "logcollector", "type": "file" } { "log": { "file": { "path": "string" } }, "tags": ["string"], "event": { "original": "string", "ingested": "string", "module": "string", "provider": "string" } } { "module": "inventory", "type": "package" } { "log": { "file": { "path": "string" } }, "tags": ["string"], "event": { "original": "string", "ingested": "string", "module": "string", "provider": "string" } } ```Stateful
```json { "agent": { "id": "2887e1cf-9bf2-431a-b066-a46860080f56", "name": "agent1", "type": "endpoint", "version": "5.0.0", "groups": ["group1", "group2"], "host": { "hostname": "myhost", "os": { "name": "Amazon Linux 2" "platform": "Linux" }, "ip": ["192.168.1.2"], "architecture": "x86_64" } } } { "module": "inventory", "type": "package", "operation": "modified", "id": "lskdjf023984902358" } { "scan_time": "2024-10-28T18:26:10.634Z", "package": { "architecture": "string", "description": "string", "installed": "2024-10-28T18:26:10.634Z", "name": "string", "path": "string", "size": 0, "type": "string", "version": "string" } } { "module": "inventory", "type": "network", "operation": "add", "id": "lskdjf023984902358" } { "scan_time": "2024-10-28T18:26:10.634Z", "package": { "architecture": "string", "description": "string", "installed": "2024-10-28T18:26:10.634Z", "name": "string",{ "agent": { "uuid": "UUID", "groups": [ ], "os": "Amazon Linux 2", "platform": "Linux", "type": "Endpoint", "version": "5.0.0", "ip": "192.168.1.2" } } "path": "string", "size": 0, "type": "string", "version": "string" } } { "module": "inventory", "type": "network", "operation": "delete", "id": "asdfsdfkdsj98237498325" } ```Tasks
296
293
295
297
298
299
284
294