Release 4.8.0 - RC 1 - Footprint Metrics - ALL-EXCEPT-SYSCOLLECTOR (2.5d)

[wazuhci]

Footprint metrics information

Main release stage issue #	#23246
Main footprint metrics issue #	#23254
Version	4.8.0
Release stage #	RC 1
Tag	https://github.com/wazuh/wazuh/tree/v4.8.0-rc1

Stress test documentation

Packages used

• Repository: packages-dev.wazuh.com

• Package path: pre-release

• Package revision: 1

• Jenkins build: https://ci.wazuh.info/job/Test_stress/5119/

---

Manager

+

Plots

                  

• Logs and configuration

  [ossec_Test_stress_B5119_manager_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_manager_centos/logs/ossec_Test_stress_B5119_manager_2024-05-06.zip)
• CSV

  [monitor-manager-Test_stress_B5119_manager-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_manager_centos/data/monitor-manager-Test_stress_B5119_manager-pre-release.csv) [Test_stress_B5119_manager_analysisd_events.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_manager_centos/data/Test_stress_B5119_manager_analysisd_events.csv) [Test_stress_B5119_manager_analysisd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_manager_centos/data/Test_stress_B5119_manager_analysisd_state.csv) [Test_stress_B5119_manager_remoted_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_manager_centos/data/Test_stress_B5119_manager_remoted_state.csv)

Centos agent

+

Plots

                

• Logs and configuration

  [ossec_Test_stress_B5119_centos_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_centos/logs/ossec_Test_stress_B5119_centos_2024-05-06.zip)
• CSV

  [monitor-agent-Test_stress_B5119_centos-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_centos/data/monitor-agent-Test_stress_B5119_centos-pre-release.csv) [Test_stress_B5119_centos_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_centos/data/Test_stress_B5119_centos_agentd_state.csv)

Ubuntu agent

+

Plots

                

• Logs and configuration

  [ossec_Test_stress_B5119_ubuntu_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_ubuntu/logs/ossec_Test_stress_B5119_ubuntu_2024-05-06.zip)
• CSV

  [monitor-agent-Test_stress_B5119_ubuntu-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_ubuntu/data/monitor-agent-Test_stress_B5119_ubuntu-pre-release.csv) [Test_stress_B5119_ubuntu_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_ubuntu/data/Test_stress_B5119_ubuntu_agentd_state.csv)

Windows agent

+

Plots

              

• Logs and configuration

  [ossec_Test_stress_B5119_windows_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_windows/logs/ossec_Test_stress_B5119_windows_2024-05-06.zip)
• CSV

  [monitor-winagent-Test_stress_B5119_windows-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_windows/data/monitor-winagent-Test_stress_B5119_windows-pre-release.csv) [Test_stress_B5119_windows_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5119-3600m/B5119_agent_windows/data/Test_stress_B5119_windows_agentd_state.csv)

macOS agent

+

Plots

• Logs and configuration
• CSV

Solaris agent

+

Plots

• Logs and configuration
• CSV

---

Plots compared to https://github.com/wazuh/wazuh/issues/23165

Conclusions 🔴

New Issues

• https://github.com/wazuh/wazuh/issues/23304

Known Issues

• https://github.com/wazuh/wazuh-jenkins/issues/4469
• https://github.com/wazuh/wazuh-jenkins/issues/4481
• https://github.com/wazuh/wazuh-jenkins/issues/4867
• https://github.com/wazuh/wazuh-jenkins/issues/5287
• https://github.com/wazuh/wazuh-jenkins/issues/6368
• https://github.com/wazuh/wazuh/issues/9311
• https://github.com/wazuh/wazuh/issues/12074
• https://github.com/wazuh/wazuh/issues/18995
• https://github.com/wazuh/wazuh/issues/22190
• https://github.com/wazuh/wazuh/issues/22565

[rafabailon]

Analysis Report

Graph Report 🔴

Manager

- **Disk, Read_Ops, Write_Ops:** Decrease in `wazuh-modulesd` - **Disk_Read:** Decrease in `wazuh-monitord`, `wazuh-syscheckd` and `wazuh-modulesd` - **PSS, RSS_MAXMIN, RSS, USS:** Decrease in `wazuh-analysisd` and Increase in `wazuh-modulesd` - **VMS:** Increase in `wazuh-remoted`, `wazuh-modulesd` and `wazuh-db` - **Decoded_Events, EDPS:** Decrease in `logcollector` - **Number_of_events:** Decrease in `events` > Known Issue: https://github.com/wazuh/wazuh/issues/22190 > New Issue: https://github.com/wazuh/wazuh/issues/23304

CentOS

- **Disk, Disk_Written, PSS, Read_Ops, RSS_MAXMIN, RSS, USS:** Decrease in `wazuh-modulesd` - **Disk_Read:** Decrease in `wazuh-syscheckd` - **VMS:** Increase in `wazuh-logcollector`, `wazuh-syscheckd`, and `wazuh-agentd` - **Write_Ops:** Decrease in `wazuh-agentd` - **AgentD_Number_of_events_buffered:** Decrease in `events`

Ubuntu

- **Disk, Disk_Read, Disk_Written:** Decrease in `wazuh-modulesd` - **VMS:** Increase in `wazuh-logcollector`, `wazuh-syscheckd`, and `wazuh-agentd` - **Write_Ops:** Decrease in `wazuh-agentd` - **AgentD_Number_of_events_buffered:** Decrease in `events`

Windows

- **Disk:** Increase in `agent` - **Disk_Read:** Decrease in `agent`

Log Report 🟡

Centos

- Expected in Stress Test ```console 2024/05/06 00:00:11 sca WARNING: Interval overtaken. 2024/05/06 00:00:24 wazuh-agentd WARNING: Agent buffer is full: Events may be lost. 2024/05/06 00:16:10 wazuh-agentd WARNING: Agent buffer is flooded: Producing too many events. 2024/05/03 21:21:13 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:21:21 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/03 21:22:00 wazuh-syscheckd WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost dataº. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287 ```console 2024/05/03 22:14:13 wazuh-syscheckd WARNING: (6922): Cannot open '/tmp/syscheck_test/directories/dir4235': No such file or directory ``` - Reported in https://github.com/wazuh/wazuh/issues/9311 ```console 2024/05/03 15:19:53 wazuh-modulesd:oscap ERROR: Internal error. Exiting... ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:19:30 wazuh-agentd ERROR: (1137): Lost connection with manager. Setting lock. 2024/05/03 15:19:30 wazuh-agentd ERROR: (1216): Unable to connect to '[172.31.10.40]:1514/tcp': 'Connection refused'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/06 00:00:36 wazuh-modulesd:ciscat ERROR: Report result file 'tmp/ciscat-report.txt' missing: No such file or directory 2024/05/06 00:00:36 wazuh-modulesd:ciscat ERROR: Failed reading scan results for policy '/var/ossec/wodles/cis-cat/benchmarks/CIS_Google_Chrome_Benchmark_v1.2.0-xccdf.xml' ```

Windows

- Expected in Stress Test ```console 2024/05/06 00:01:36 wazuh-agent WARNING: Agent buffer at 90 %. 2024/05/06 00:01:37 wazuh-agent WARNING: Agent buffer is full: Events may be lost. 2024/05/06 00:04:29 sca WARNING: Interval overtaken. 2024/05/06 01:26:12 wazuh-modulesd:ciscat WARNING: Interval overtaken. 2024/05/03 21:21:16 wazuh-agent WARNING: (6906): Real time process: no data. Probably buffer overflow. 2024/05/03 21:21:40 wazuh-agent WARNING: (1960): File limit has been reached (200). 2024/05/03 21:25:59 wazuh-agent WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. ``` - Reported in https://github.com/wazuh/wazuh/issues/12074 ```console 2024/05/05 15:21:19 wazuh-agent ERROR: (6613): Real time Windows callback process: 'Access is denied.' (5). ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287 ```console 2024/05/05 13:35:20 wazuh-agent ERROR: (1103): Could not open file 'C:\tmp\logcollector_test\date-2024-05-05-160-924.log' due to [(2)-(No such file or directory)]. [2024-05-04_05:37:32] [ERROR] (create_delete): files\fimStress.3415320 file cannot be deleted. ``` - Reported in https://github.com/wazuh/wazuh/issues/18995 ```console 2024/05/03 21:24:56 wazuh-agent ERROR: (6716): Could not open handle for 'c:\tmp\syscheck_test\files\fimstress.28210'. Error code: 2 2024/05/03 21:24:56 wazuh-agent WARNING: At get_user(c:\tmp\syscheck_test\files\fimstress.28210): CreateFile(): The system cannot find the file specified. (2) 2024/05/03 22:08:37 wazuh-agent WARNING: (6922): Cannot open 'c:\tmp\syscheck_test\directories\dir2647': No such file or directory ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:19:35 wazuh-agent ERROR: (1216): Unable to connect to '[172.31.10.40]:1514/tcp': 'No connection could be made because the target machine actively refused it.'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/06 00:26:12 wazuh-modulesd:ciscat ERROR: Timeout expired executing 'C:\cis-cat\benchmarks\CIS_Microsoft_Windows_Server_2016_Benchmark_v1.0.0-xccdf.xml'. 2024/05/03 15:20:09 wazuh-modulesd:ciscat ERROR: Report result file 'tmp\ciscat-report.txt' missing: No such file or directory ```

Ubuntu

- Expected in Stress Test ```console 2024/05/06 00:00:12 wazuh-agentd WARNING: Agent buffer is full: Events may be lost. 2024/05/06 00:01:03 sca WARNING: Interval overtaken. 2024/05/03 21:21:29 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:21:38 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/03 21:22:00 wazuh-syscheckd WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data. 2024/05/04 05:00:58 wazuh-agentd WARNING: Agent buffer is flooded: Producing too many events. ``` - Reported in https://github.com/wazuh/wazuh/issues/9311 ```console 2024/05/03 15:19:52 wazuh-modulesd:oscap ERROR: Internal error. Exiting... ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:19:36 wazuh-agentd ERROR: (1216): Unable to connect to '[172.31.10.40]:1514/tcp': 'Connection refused'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/03 15:19:19 wazuh-modulesd:ciscat ERROR: CIS-CAT tool not found at '/var/ossec/wodles/cis-cat'. ```

Manager

- Expected in Stress Test ```console 2024/05/06 00:00:15 sca WARNING: Interval overtaken. 2024/05/03 15:18:34 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/03 16:53:51 wazuh-remoted WARNING: Message queue is full (10). Events may be lost. 2024/05/03 17:19:53 wazuh-analysisd WARNING: Security Configuration Assessment decoder queue is full. 2024/05/03 21:21:42 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:21:42 wazuh-analysisd WARNING: Input queue is full. 2024/05/03 21:22:07 wazuh-syscheckd WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287 ```console 2024/05/04 03:36:53 wazuh-syscheckd WARNING: (6922): Cannot open '/tmp/syscheck_test/directories/dir4041': No such file or directory ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:19:47 wazuh-remoted WARNING: Agent key already in use: agent ID '002' ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/6368 ```console 2024/05/03 15:19:45 wazuh-modulesd:vulnerability-scanner WARNING: The 'feed-update-interval' option at module 'vulnerability-detection' must be at least 1 hour. Automatically set to 60 minutes. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/03 15:19:44 wazuh-modulesd:ciscat WARNING: No evals defined. Exiting... ``` - Reported in https://github.com/wazuh/wazuh/issues/22565 ```console 2024/05/03 15:16:27 indexer-connector WARNING: No username and password found in the keystore, using default values. 2024/05/03 15:16:27 indexer-connector WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities', retrying until the connection is successful. 2024/05/03 15:17:40 indexer-connector WARNING: Failed to sync agent '000' with the indexer. ``` - Reported in https://github.com/wazuh/wazuh/issues/9311 ```console 2024/05/03 15:19:44 wazuh-modulesd:oscap ERROR: Internal error. Exiting... ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4481 ```console 2024/05/06 00:47:57 wazuh-modulesd:azure-logs ERROR: azure-activity: Returned error code: '1'. ```

[MARCOSD4]

LGTM

[juliamagan]

LGTM