wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
9.34k stars 1.48k forks source link

Release 4.8.0 - RC 1 - Footprint Metrics - LOGCOLLECTOR,SCA (2.5d) #23294

Closed wazuhci closed 1 week ago

wazuhci commented 1 week ago

Footprint metrics information

Main release stage issue # #23246
Main footprint metrics issue # #23254
Version 4.8.0
Release stage # RC 1
Tag https://github.com/wazuh/wazuh/tree/v4.8.0-rc1

Stress test documentation

Packages used


Manager +
Plots ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_CPU.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_Disk.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_Disk_Read.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_Disk_Written.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_FD.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_PSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_Read_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_RSS_MAXMIN.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_RSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_SWAP.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_USS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_VMS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/monitor-manager-Test_stress_B5110_manager-pre-release_Write_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/Test_stress_B5110_manager_analysisd_events_Decoded_events.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/Test_stress_B5110_manager_analysisd_events_Dropped_events.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/Test_stress_B5110_manager_analysisd_events_EDPS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/Test_stress_B5110_manager_analysisd_events_Written_stats.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/Test_stress_B5110_manager_analysisd_state_Number_Events.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/plots/Test_stress_B5110_manager_analysisd_state_Queues_state.png)
  • Logs and configuration [ossec_Test_stress_B5110_manager_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/logs/ossec_Test_stress_B5110_manager_2024-05-06.zip)
  • CSV [monitor-manager-Test_stress_B5110_manager-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/data/monitor-manager-Test_stress_B5110_manager-pre-release.csv) [Test_stress_B5110_manager_analysisd_events.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/data/Test_stress_B5110_manager_analysisd_events.csv) [Test_stress_B5110_manager_analysisd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/data/Test_stress_B5110_manager_analysisd_state.csv) [Test_stress_B5110_manager_remoted_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_manager_centos/data/Test_stress_B5110_manager_remoted_state.csv)

Centos agent +
Plots ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_CPU.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_Disk.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_Disk_Read.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_Disk_Written.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_FD.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_PSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_Read_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_RSS_MAXMIN.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_RSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_SWAP.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_USS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_VMS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/monitor-agent-Test_stress_B5110_centos-pre-release_Write_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/Test_stress_B5110_centos_agentd_state_AgentD_Number_of_events_buffered.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/Test_stress_B5110_centos_agentd_state_AgentD_Number_of_generated_events.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/Test_stress_B5110_centos_agentd_state_AgentD_Number_of_messages.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/plots/Test_stress_B5110_centos_agentd_state_AgentD_Status.png)
  • Logs and configuration [ossec_Test_stress_B5110_centos_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/logs/ossec_Test_stress_B5110_centos_2024-05-06.zip)
  • CSV [monitor-agent-Test_stress_B5110_centos-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/data/monitor-agent-Test_stress_B5110_centos-pre-release.csv) [Test_stress_B5110_centos_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_centos/data/Test_stress_B5110_centos_agentd_state.csv)

Ubuntu agent +
Plots ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_CPU.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_Disk.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_Disk_Read.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_Disk_Written.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_FD.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_PSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_Read_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_RSS_MAXMIN.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_RSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_SWAP.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_USS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_VMS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/monitor-agent-Test_stress_B5110_ubuntu-pre-release_Write_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/Test_stress_B5110_ubuntu_agentd_state_AgentD_Number_of_events_buffered.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/Test_stress_B5110_ubuntu_agentd_state_AgentD_Number_of_generated_events.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/Test_stress_B5110_ubuntu_agentd_state_AgentD_Number_of_messages.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/plots/Test_stress_B5110_ubuntu_agentd_state_AgentD_Status.png)
  • Logs and configuration [ossec_Test_stress_B5110_ubuntu_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/logs/ossec_Test_stress_B5110_ubuntu_2024-05-06.zip)
  • CSV [monitor-agent-Test_stress_B5110_ubuntu-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/data/monitor-agent-Test_stress_B5110_ubuntu-pre-release.csv) [Test_stress_B5110_ubuntu_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_ubuntu/data/Test_stress_B5110_ubuntu_agentd_state.csv)

Windows agent +
Plots ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_CPU.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_Disk.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_Disk_Read.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_Disk_Written.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_Handles.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_Read_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_RSS_MAXMIN.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_RSS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_USS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_VMS.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/monitor-winagent-Test_stress_B5110_windows-pre-release_Write_Ops.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/Test_stress_B5110_windows_agentd_state_AgentD_Number_of_events_buffered.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/Test_stress_B5110_windows_agentd_state_AgentD_Number_of_generated_events.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/Test_stress_B5110_windows_agentd_state_AgentD_Number_of_messages.png) ![](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/plots/Test_stress_B5110_windows_agentd_state_AgentD_Status.png)
  • Logs and configuration [ossec_Test_stress_B5110_windows_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/logs/ossec_Test_stress_B5110_windows_2024-05-06.zip)
  • CSV [monitor-winagent-Test_stress_B5110_windows-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/data/monitor-winagent-Test_stress_B5110_windows-pre-release.csv) [Test_stress_B5110_windows_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5110-3600m/B5110_agent_windows/data/Test_stress_B5110_windows_agentd_state.csv)

macOS agent +
Plots
  • Logs and configuration
  • CSV

Solaris agent +
Plots
  • Logs and configuration
  • CSV

Conclusion :red_circle:

Plots compared to 4.7.4-RC2

Logs :yellow_circle:

Known issues:

Graphs :red_circle:

Known issue:

New issue:

santipadilla commented 1 week ago

Analysis Report

Logs :yellow_circle:

Manager - Expected in Stress Test ``` 2024/05/06 00:00:20 sca WARNING: Interval overtaken. 2024/05/03 15:19:42 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/03 21:22:01 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:22:03 wazuh-analysisd WARNING: Input queue is full. 2024/05/03 21:23:34 wazuh-remoted WARNING: Message queue is full (10). Events may be lost. 2024/05/03 22:11:53 wazuh-analysisd WARNING: Security Configuration Assessment decoder queue is full. ``` - Reported in: https://github.com/wazuh/wazuh/issues/22565 - Known issue ``` 2024/05/03 15:17:35 indexer-connector WARNING: No username and password found in the keystore, using default values. 2024/05/03 15:17:35 indexer-connector WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities', retrying until the connection is successful. 2024/05/03 15:18:47 indexer-connector WARNING: Failed to sync agent '000' with the indexer. ```
Windows - Expected in Stress Test ``` 2024/05/06 00:05:30 sca WARNING: Interval overtaken. 2024/05/04 00:00:10 wazuh-agent WARNING: Agent buffer is full: Events may be lost. 2024/05/04 00:00:52 wazuh-agent WARNING: (1960): File limit has been reached (200). 2024/05/04 00:01:11 wazuh-agent WARNING: Agent buffer at 90 %. 2024/05/03 21:51:59 wazuh-agent WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. ```
centOS - Expected in Stress Test ``` 2024/05/06 00:01:44 sca WARNING: Interval overtaken. 2024/05/06 00:06:03 wazuh-agentd WARNING: Agent buffer is full: Events may be lost. 2024/05/03 21:21:38 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:21:45 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/03 21:25:07 wazuh-agentd WARNING: Agent buffer is flooded: Producing too many events. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 - Known issue ``` 2024/05/03 15:20:11 wazuh-agentd ERROR: (1137): Lost connection with manager. Setting lock. 2024/05/03 15:20:11 wazuh-agentd ERROR: (1216): Unable to connect to '[172.31.3.129]:1514/tcp': 'Connection refused'. 2024/05/03 15:20:11 wazuh-modulesd WARNING: Process locked due to agent is offline. Waiting for connection... ```
Ubuntu - Expected in Stress Test ``` 2024/05/06 00:00:11 wazuh-agentd WARNING: Agent buffer is full: Events may be lost. 2024/05/06 00:01:37 sca WARNING: Interval overtaken. 2024/05/03 21:22:11 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:22:18 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/05 15:06:06 wazuh-agentd WARNING: Agent buffer is flooded: Producing too many events. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 - Known issue ``` 2024/05/03 15:20:17 wazuh-agentd ERROR: (1216): Unable to connect to '[172.31.3.129]:1514/tcp': 'Connection refused'. ```

Graphs :red_circle:

Manager

Disk: decrease of modulesd PSS, RSS, USS: increase of modulesd

VMS: small increase of remoted and db

centOS

Disk: decrease of modulesd PSS, RSS, USS: small increase of modulesd

Ubuntu

Disk: decrease of modulesd Disk_read: small increase of modulesd VMS: small decrease of modulesd

Windows

No abnormalities were found

Rebits commented 1 week ago

Graphs

santipadilla commented 1 week ago

Done @Rebits, thanks!

Rebits commented 1 week ago

LGTM

juliamagan commented 1 week ago

LGTM