Release 4.8.0 - RC 1 - Footprint Metrics - ALL (2.5d)

[wazuhci]

Footprint metrics information

Main release stage issue #	#23246
Main footprint metrics issue #	#23254
Version	4.8.0
Release stage #	RC 1
Tag	https://github.com/wazuh/wazuh/tree/v4.8.0-rc1

Stress test documentation

Packages used

• Repository: packages-dev.wazuh.com

• Package path: pre-release

• Package revision: 1

• Jenkins build: https://ci.wazuh.info/job/Test_stress/5113/

---

Manager

+

Plots

                  

• Logs and configuration

  [ossec_Test_stress_B5113_manager_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_manager_centos/logs/ossec_Test_stress_B5113_manager_2024-05-06.zip)
• CSV

  [monitor-manager-Test_stress_B5113_manager-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_manager_centos/data/monitor-manager-Test_stress_B5113_manager-pre-release.csv) [Test_stress_B5113_manager_analysisd_events.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_manager_centos/data/Test_stress_B5113_manager_analysisd_events.csv) [Test_stress_B5113_manager_analysisd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_manager_centos/data/Test_stress_B5113_manager_analysisd_state.csv) [Test_stress_B5113_manager_remoted_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_manager_centos/data/Test_stress_B5113_manager_remoted_state.csv)

Centos agent

+

Plots

                

• Logs and configuration

  [ossec_Test_stress_B5113_centos_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_centos/logs/ossec_Test_stress_B5113_centos_2024-05-06.zip)
• CSV

  [monitor-agent-Test_stress_B5113_centos-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_centos/data/monitor-agent-Test_stress_B5113_centos-pre-release.csv) [Test_stress_B5113_centos_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_centos/data/Test_stress_B5113_centos_agentd_state.csv)

Ubuntu agent

+

Plots

                

• Logs and configuration

  [ossec_Test_stress_B5113_ubuntu_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_ubuntu/logs/ossec_Test_stress_B5113_ubuntu_2024-05-06.zip)
• CSV

  [monitor-agent-Test_stress_B5113_ubuntu-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_ubuntu/data/monitor-agent-Test_stress_B5113_ubuntu-pre-release.csv) [Test_stress_B5113_ubuntu_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_ubuntu/data/Test_stress_B5113_ubuntu_agentd_state.csv)

Windows agent

+

Plots

              

• Logs and configuration

  [ossec_Test_stress_B5113_windows_2024-05-06.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_windows/logs/ossec_Test_stress_B5113_windows_2024-05-06.zip)
• CSV

  [monitor-winagent-Test_stress_B5113_windows-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_windows/data/monitor-winagent-Test_stress_B5113_windows-pre-release.csv) [Test_stress_B5113_windows_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.8.0/B5113-3600m/B5113_agent_windows/data/Test_stress_B5113_windows_agentd_state.csv)

macOS agent

+

Plots

• Logs and configuration
• CSV

Solaris agent

+

Plots

• Logs and configuration
• CSV

Conclusion :red_circle:

Plots compared to https://github.com/wazuh/wazuh/issues/23174

Graphs :red_circle:

New issue:

• https://github.com/wazuh/wazuh/issues/23304

Logs :red_circle:

New issue:

• https://github.com/wazuh/wazuh/issues/23301

Known issues:

• https://github.com/wazuh/wazuh-jenkins/issues/5287
• https://github.com/wazuh/wazuh/issues/12074
• https://github.com/wazuh/wazuh/issues/18995
• https://github.com/wazuh/wazuh-jenkins/issues/4867
• https://github.com/wazuh/wazuh-jenkins/issues/4469
• https://github.com/wazuh/wazuh/issues/9311
• https://github.com/wazuh/wazuh/issues/22565
• https://github.com/wazuh/wazuh-jenkins/issues/4481
• https://github.com/wazuh/wazuh-jenkins/issues/6368

[MARCOSD4]

Analysis Report

Graph Report :red_circle:

Manager

- **Disk:** decrease in `wazuh-modulesd` and `wazuh-db`. - **FD, PSS, RSS_MAXMIN, RSS, USS:** increase in wazuh-modulesd: - New issue: https://github.com/wazuh/wazuh/issues/23304 - **PSS, RSS_MAXMIN, RSS, USS:** small increase in `wazuh-analysisd`. - **VMS:** small increase in `wazuh-remoted`, `wazuh-db` and `wazuh-modulesd`.

CentOS

- **Disk:** decrease in `wazuh-moduesd`. - **PSS, RSS_MAXMIN, RSS, USS:** small decrease in `wazuh-modulesd`.

Ubuntu

- No Abnormalities Detected

Windows

- No Abnormalities Detected

Log Report :red_circle:

Windows

- Expected in Stress Tests ```console 2024/05/05 00:00:11 wazuh-agent WARNING: Agent buffer is full: Events may be lost. 2024/05/05 00:00:12 wazuh-agent WARNING: (6906): Real time process: no data. Probably buffer overflow. 2024/05/05 00:00:19 wazuh-agent WARNING: (1960): File limit has been reached (200). 2024/05/05 00:01:19 wazuh-agent WARNING: Agent buffer at 90 %. 2024/05/05 00:06:50 sca WARNING: Interval overtaken. 2024/05/05 01:26:46 wazuh-modulesd:ciscat WARNING: Interval overtaken. 2024/05/03 21:22:24 wazuh-agent WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287 ```console [2024-05-03_23:09:03] [ERROR] (create_delete): files\fimStress.735214 file cannot be deleted. ``` - Reported in https://github.com/wazuh/wazuh/issues/12074 ```console 2024/05/05 15:21:52 wazuh-agent ERROR: (6613): Real time Windows callback process: 'Access is denied.' (5). ``` - Reported in https://github.com/wazuh/wazuh/issues/18995 ```console 2024/05/05 00:00:32 wazuh-agent ERROR: (6716): Could not open handle for 'c:\tmp\syscheck_test\files\fimstress.10948588'. Error code: 2 2024/05/05 00:00:32 wazuh-agent WARNING: At get_user(c:\tmp\syscheck_test\files\fimstress.10948588): CreateFile(): The system cannot find the file specified. (2) 2024/05/05 00:18:04 wazuh-agent WARNING: (6922): Cannot open 'c:\tmp\syscheck_test\directories\dir1362': No such file or directory ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:20:10 wazuh-agent ERROR: (1216): Unable to connect to '[172.31.4.126]:1514/tcp': 'No connection could be made because the target machine actively refused it.'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/05 00:26:47 wazuh-modulesd:ciscat ERROR: Timeout expired executing 'C:\cis-cat\benchmarks\CIS_Microsoft_Windows_Server_2016_Benchmark_v1.0.0-xccdf.xml'. 2024/05/03 15:20:55 wazuh-modulesd:ciscat ERROR: Report result file 'tmp\ciscat-report.txt' missing: No such file or directory ```

Centos

- Expected in Stress Tests ```console 2024/05/06 00:00:22 sca WARNING: Interval overtaken. 2024/05/06 00:00:33 wazuh-agentd WARNING: Agent buffer is full: Events may be lost. 2024/05/06 00:04:06 wazuh-agentd WARNING: Agent buffer is flooded: Producing too many events. 2024/05/05 00:00:30 wazuh-syscheckd WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data. 2024/05/05 00:07:55 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/05 05:18:14 wazuh-modulesd:ciscat WARNING: Interval overtaken. 2024/05/03 21:22:00 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287 ```console 2024/05/04 02:36:24 wazuh-syscheckd WARNING: (6922): Cannot open '/tmp/syscheck_test/directories/dir862': No such file or directory ``` - Reported in https://github.com/wazuh/wazuh/issues/9311 ```console 2024/05/03 15:20:22 wazuh-modulesd:oscap ERROR: Internal error. Exiting... ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:20:04 wazuh-agentd ERROR: (1137): Lost connection with manager. Setting lock. 2024/05/03 15:20:04 wazuh-agentd ERROR: (1216): Unable to connect to '[172.31.4.126]:1514/tcp': 'Connection refused'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/06 00:00:17 wazuh-modulesd:ciscat ERROR: Report result file 'tmp/ciscat-report.txt' missing: No such file or directory 2024/05/06 00:00:17 wazuh-modulesd:ciscat ERROR: Failed reading scan results for policy '/var/ossec/wodles/cis-cat/benchmarks/CIS_Google_Chrome_Benchmark_v1.2.0-xccdf.xml' ```

Ubuntu

- Expected in Stress Tests ```console 2024/05/06 00:00:12 wazuh-agentd WARNING: Agent buffer is full: Events may be lost. 2024/05/06 00:01:52 sca WARNING: Interval overtaken. 2024/05/06 02:31:35 wazuh-agentd WARNING: Agent buffer is flooded: Producing too many events. 2024/05/05 00:00:24 wazuh-syscheckd WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data. 2024/05/05 00:09:16 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/03 21:22:16 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. ``` - Reported in https://github.com/wazuh/wazuh/issues/9311 ```console 2024/05/03 15:20:24 wazuh-modulesd:oscap ERROR: Internal error. Exiting... ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:20:11 wazuh-agentd ERROR: (1216): Unable to connect to '[172.31.4.126]:1514/tcp': 'Connection refused'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/03 15:19:54 wazuh-modulesd:ciscat ERROR: CIS-CAT tool not found at '/var/ossec/wodles/cis-cat'. ```

Manager

- Expected in Stress Tests ```console 2024/05/06 00:00:21 sca WARNING: Interval overtaken. 2024/05/05 00:02:12 wazuh-syscheckd WARNING: Real-time inotify kernel queue is full. Some events may be lost. Next scheduled scan will recover lost data. 2024/05/05 00:07:49 wazuh-logcollector WARNING: (1960): File limit has been reached (1000). Please reduce the number of files or increase "logcollector.max_files". 2024/05/05 06:01:46 wazuh-analysisd WARNING: Syscollector decoder queue is full. 2024/05/03 15:19:19 wazuh-logcollector WARNING: Target 'agent' message queue is full (1024). Log lines may be lost. 2024/05/03 17:22:37 wazuh-remoted WARNING: Message queue is full (10). Events may be lost. 2024/05/03 21:22:21 wazuh-analysisd WARNING: Input queue is full. 2024/05/03 23:01:03 wazuh-analysisd WARNING: Security Configuration Assessment decoder queue is full. ``` - New issue: https://github.com/wazuh/wazuh/issues/23301 ```console 2024/05/06 00:07:34 wazuh-db ERROR: Socket 56 error: Broken pipe (32) 2024/05/06 00:07:34 wazuh-db ERROR: at run_worker(): wnotify_add(56): Bad file descriptor (9) ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4867 ```console 2024/05/03 15:20:22 wazuh-remoted WARNING: Agent key already in use: agent ID '002' ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/4469 ```console 2024/05/03 15:20:20 wazuh-modulesd:ciscat WARNING: No evals defined. Exiting... ``` - Reported in https://github.com/wazuh/wazuh/issues/22565 ```console 2024/05/03 15:17:12 indexer-connector WARNING: No username and password found in the keystore, using default values. 2024/05/03 15:17:12 indexer-connector WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities', retrying until the connection is successful. 2024/05/03 15:18:25 indexer-connector WARNING: Failed to sync agent '000' with the indexer. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/5287 ```console 2024/05/05 00:11:44 wazuh-syscheckd WARNING: (6922): Cannot open '/tmp/syscheck_test/directories/dir3746': No such file or directory ``` - Reported in https://github.com/wazuh/wazuh/issues/9311 ```console 2024/05/03 15:20:20 wazuh-modulesd:oscap ERROR: Internal error. Exiting... ``` - Reported inhttps://github.com/wazuh/wazuh-jenkins/issues/4481 ```console 2024/05/06 01:19:33 wazuh-modulesd:azure-logs ERROR: azure-activity: Returned error code: '1'. ``` - Reported in https://github.com/wazuh/wazuh-jenkins/issues/6368 ```console 2024/05/03 15:20:20 wazuh-modulesd:vulnerability-scanner WARNING: The 'feed-update-interval' option at module 'vulnerability-detection' must be at least 1 hour. Automatically set to 60 minutes. ```

[Rebits]

Logs

• The reported indexer warning https://github.com/wazuh/wazuh/issues/23303 is produced due to the known issue: https://github.com/wazuh/wazuh/issues/22565. I suggest closing this issue.

However, in order to enhance user experience, it's advisable to make warning logs more informative, especially in cases of misconfiguration such as missing key files or incorrect indexer settings. I recommend opening a new issue to address this concern

Graphics

• Dramatic increase of memory. Distinct from https://github.com/wazuh/wazuh/issues/23202, due to file descriptors graphics not showing the leak. We should open a new issue reporting the drastic increase of memory used by modules. Related to https://github.com/wazuh/wazuh/issues/22841

[Rebits]

LGTM

[juliamagan]

LGTM