search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
61 stars 30 forks source link

Functionality tests for individual sub-indices and global index approach #5302

Closed juliamagan closed 2 weeks ago

juliamagan commented 3 weeks ago

Description

This issue is intended to cover the testing process followed and obtained for https://github.com/wazuh/wazuh/issues/23058 development.

Test environment

Component Quantity Operating System CPU (cores) RAM (GB) Disk (GB)
Master 1 Ubuntu 22 4 8 50
Workers 2 Ubuntu 22 4 8 50
Agent 1 1 Ubuntu 22 2 4 30
Agent 2 1 Windows 11 2 4 30
Load Balancer 1 Ubuntu 22 4 8 50
Indexers 2 Ubuntu 22 2 4 30

[!NOTE] The load balancer is located on the master node.

Architecture Framework development package URL URL
DEB 4.8.0-python.vd.spike.deb.1
RPM 4.8.0-python.vd.spike.rpm.1

Test cases

ID Description Status
T1 Change of worker due to agent restart
T2 Change of worker due to connection loss
T3 Change of worker due to worker restart
T4 Loss and recovery of connection to a worker
T5 Change of worker due to kill of worker
T6 Restart of worker while vulnerability scanning is in progress
T7 Restart of indexer while states are being indexed
T8 Loss of connection of indexer while states are being indexed
T9 Restart of indexer while states are being indexed and agent changes worker
T10 Loss of connection of indexer while states are being indexed and agent changes worker
T11 Restart of both indexers while states are being indexed
T12 Loss of connection of both indexers while states are being indexed
T13 Installation of a vulnerable package when the agent is stopped
T14 Removal of a vulnerable package when the agent is stopped
T15 Removal of a vulnerable package and change worker with long syscollector interval
T16 Corrupt hash
T17 Deleted hash
T18 Agent with syscollector enabled later
T19 All agents conected to master node
T20 Exchange of agents and workers
T21 Agent re-registered with same name but different ID
T22 Agent re-registered with same ID but different name
T23 Restart of the master node
T24 Loss of connection with the master node
T25 While the agent is switching workers, the target worker is restarted

Status legend:

🟢 - Approved 🟡 - Approved with warnings or expected errors 🔴 - Rejected

santipadilla commented 2 weeks ago

We have used the following environment:

Master

System information
```console root@ip-172-31-40-132:/home/ubuntu# cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ```
Manager version
```console root@ip-172-31-40-132:/home/ubuntu# /var/ossec/bin/wazuh-control info WAZUH_VERSION="v4.8.0" WAZUH_REVISION="40807" WAZUH_TYPE="server" ```
Manager status
```console root@ip-172-31-40-132:/home/ubuntu# systemctl status wazuh-manager -l ● wazuh-manager.service - Wazuh manager Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 14:24:03 UTC; 16h ago Tasks: 171 (limit: 18969) Memory: 246.4M CPU: 8min 56.810s CGroup: /system.slice/wazuh-manager.service ├─53519 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─53520 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─53523 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─53526 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─53567 /var/ossec/bin/wazuh-authd ├─53583 /var/ossec/bin/wazuh-db ├─53608 /var/ossec/bin/wazuh-execd ├─53619 /var/ossec/bin/wazuh-analysisd ├─53631 /var/ossec/bin/wazuh-syscheckd ├─53697 /var/ossec/bin/wazuh-remoted ├─53732 /var/ossec/bin/wazuh-logcollector ├─53753 /var/ossec/bin/wazuh-monitord └─53804 /var/ossec/bin/wazuh-modulesd Apr 29 14:23:56 ip-172-31-40-132 env[53463]: Started wazuh-analysisd... Apr 29 14:23:57 ip-172-31-40-132 env[53463]: Started wazuh-syscheckd... Apr 29 14:23:58 ip-172-31-40-132 env[53463]: Started wazuh-remoted... Apr 29 14:23:59 ip-172-31-40-132 env[53463]: Started wazuh-logcollector... Apr 29 14:24:00 ip-172-31-40-132 env[53463]: Started wazuh-monitord... Apr 29 14:24:00 ip-172-31-40-132 env[53802]: 2024/04/29 14:24:00 wazuh-modulesd:router: INFO: Loaded ro> Apr 29 14:24:00 ip-172-31-40-132 env[53802]: 2024/04/29 14:24:00 wazuh-modulesd:content_manager: INFO: > Apr 29 14:24:01 ip-172-31-40-132 env[53463]: Started wazuh-modulesd... Apr 29 14:24:03 ip-172-31-40-132 env[53463]: Completed. Apr 29 14:24:03 ip-172-31-40-132 systemd[1]: Started Wazuh manager. ```

Workers

System information
```console root@ip-172-31-35-68:/home/ubuntu# cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ```
Manager version
```console root@ip-172-31-35-68:/home/ubuntu# /var/ossec/bin/wazuh-control info WAZUH_VERSION="v4.8.0" WAZUH_REVISION="40807" WAZUH_TYPE="server" ```
Worker 1 status
```console root@ip-172-31-35-68:/home/ubuntu# systemctl status wazuh-manager -l ● wazuh-manager.service - Wazuh manager Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 10:08:06 UTC; 21h ago Tasks: 171 (limit: 18969) Memory: 6.1G CPU: 13min 24.877s CGroup: /system.slice/wazuh-manager.service ├─58668 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─58707 /var/ossec/bin/wazuh-authd ├─58723 /var/ossec/bin/wazuh-db ├─58748 /var/ossec/bin/wazuh-execd ├─58759 /var/ossec/bin/wazuh-analysisd ├─58762 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─58765 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─58768 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─58791 /var/ossec/bin/wazuh-syscheckd ├─58858 /var/ossec/bin/wazuh-remoted ├─58891 /var/ossec/bin/wazuh-logcollector ├─58911 /var/ossec/bin/wazuh-monitord └─58933 /var/ossec/bin/wazuh-modulesd Apr 29 10:07:58 ip-172-31-35-68 env[58586]: Started wazuh-analysisd... Apr 29 10:07:59 ip-172-31-35-68 env[58586]: Started wazuh-syscheckd... Apr 29 10:08:01 ip-172-31-35-68 env[58586]: Started wazuh-remoted... Apr 29 10:08:02 ip-172-31-35-68 env[58586]: Started wazuh-logcollector... Apr 29 10:08:03 ip-172-31-35-68 env[58586]: Started wazuh-monitord... Apr 29 10:08:03 ip-172-31-35-68 env[58930]: 2024/04/29 10:08:03 wazuh-modulesd:router: INFO: Loaded rout> Apr 29 10:08:03 ip-172-31-35-68 env[58930]: 2024/04/29 10:08:03 wazuh-modulesd:content_manager: INFO: Lo> Apr 29 10:08:04 ip-172-31-35-68 env[58586]: Started wazuh-modulesd... Apr 29 10:08:06 ip-172-31-35-68 env[58586]: Completed. Apr 29 10:08:06 ip-172-31-35-68 systemd[1]: Started Wazuh manager. ```
Worker 2 status
```console root@ip-172-31-39-229:/home/ubuntu# systemctl status wazuh-manager -l ● wazuh-manager.service - Wazuh manager Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 10:18:19 UTC; 21h ago Tasks: 171 (limit: 18969) Memory: 6.1G CPU: 13min 53.172s CGroup: /system.slice/wazuh-manager.service ├─48087 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─48126 /var/ossec/bin/wazuh-authd ├─48142 /var/ossec/bin/wazuh-db ├─48170 /var/ossec/bin/wazuh-execd ├─48174 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─48177 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─48180 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh-apid.py ├─48193 /var/ossec/bin/wazuh-analysisd ├─48208 /var/ossec/bin/wazuh-syscheckd ├─48272 /var/ossec/bin/wazuh-remoted ├─48307 /var/ossec/bin/wazuh-logcollector ├─48325 /var/ossec/bin/wazuh-monitord └─48350 /var/ossec/bin/wazuh-modulesd Apr 29 10:18:12 ip-172-31-39-229 env[47628]: Started wazuh-analysisd... Apr 29 10:18:13 ip-172-31-39-229 env[47628]: Started wazuh-syscheckd... Apr 29 10:18:14 ip-172-31-39-229 env[47628]: Started wazuh-remoted... Apr 29 10:18:15 ip-172-31-39-229 env[47628]: Started wazuh-logcollector... Apr 29 10:18:16 ip-172-31-39-229 env[47628]: Started wazuh-monitord... Apr 29 10:18:16 ip-172-31-39-229 env[48348]: 2024/04/29 10:18:16 wazuh-modulesd:router: INFO: Loaded ro> Apr 29 10:18:16 ip-172-31-39-229 env[48348]: 2024/04/29 10:18:16 wazuh-modulesd:content_manager: INFO: > Apr 29 10:18:17 ip-172-31-39-229 env[47628]: Started wazuh-modulesd... Apr 29 10:18:19 ip-172-31-39-229 env[47628]: Completed. Apr 29 10:18:19 ip-172-31-39-229 systemd[1]: Started Wazuh manager. ```

Indexers

System information
```console root@ip-172-31-42-75:/home/ubuntu# cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ```
Indexer 1 status
```console root@ip-172-31-42-75:/home/ubuntu# systemctl status wazuh-indexer -l ● wazuh-indexer.service - Wazuh-indexer Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 09:15:56 UTC; 22h ago Docs: https://documentation.wazuh.com Main PID: 6111 (java) Tasks: 64 (limit: 4589) Memory: 2.3G CPU: 13min 41.336s CGroup: /system.slice/wazuh-indexer.service └─6111 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cach> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at org.opensearch.jobscheduler.sweepe> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at org.opensearch.threadpool.Schedule> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at org.opensearch.common.util.concurr> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at org.opensearch.common.util.concurr> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at java.base/java.util.concurrent.Exe> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at java.base/java.util.concurrent.Fut> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at java.base/java.util.concurrent.Sch> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at java.base/java.util.concurrent.Thr> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at java.base/java.util.concurrent.Thr> Apr 30 00:00:26 ip-172-31-42-75 systemd-entrypoint[6111]: at java.base/java.lang.Thread.run(Thr> ```
Indexer 2 status
```console root@ip-172-31-46-237:/home/ubuntu# systemctl status wazuh-indexer -l ● wazuh-indexer.service - Wazuh-indexer Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 09:16:40 UTC; 22h ago Docs: https://documentation.wazuh.com Main PID: 14889 (java) Tasks: 69 (limit: 4589) Memory: 2.3G CPU: 14min 28.388s CGroup: /system.slice/wazuh-indexer.service └─14889 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cac> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.cluster.service.M> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.cluster.service.M> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.cluster.service.T> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.cluster.service.T> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.common.util.concu> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.common.util.concu> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at org.opensearch.common.util.concu> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at java.base/java.util.concurrent.T> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at java.base/java.util.concurrent.T> Apr 30 00:01:37 ip-172-31-46-237 systemd-entrypoint[14889]: at java.base/java.lang.Thread.run(T> ```

Ubuntu Agent

System information
```console root@ip-172-31-36-193:/home/ubuntu# cat /etc/*release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.4 LTS" PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ```
Agent version
```console root@ip-172-31-36-193:/home/ubuntu# /var/ossec/bin/wazuh-control info WAZUH_VERSION="v4.8.0" WAZUH_REVISION="40808" WAZUH_TYPE="agent" ```
Agent status
```console root@ip-172-31-36-193:/home/ubuntu# systemctl status wazuh-agent ● wazuh-agent.service - Wazuh agent Loaded: loaded (/lib/systemd/system/wazuh-agent.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2024-04-29 10:21:44 UTC; 21h ago Tasks: 32 (limit: 4589) Memory: 225.4M CPU: 1min 46.490s CGroup: /system.slice/wazuh-agent.service ├─3716 /var/ossec/bin/wazuh-execd ├─3727 /var/ossec/bin/wazuh-agentd ├─3741 /var/ossec/bin/wazuh-syscheckd ├─3755 /var/ossec/bin/wazuh-logcollector └─3772 /var/ossec/bin/wazuh-modulesd Apr 29 10:21:37 ip-172-31-36-193 systemd[1]: Starting Wazuh agent... Apr 29 10:21:37 ip-172-31-36-193 env[3074]: Starting Wazuh v4.8.0... Apr 29 10:21:38 ip-172-31-36-193 env[3074]: Started wazuh-execd... Apr 29 10:21:39 ip-172-31-36-193 env[3074]: Started wazuh-agentd... Apr 29 10:21:40 ip-172-31-36-193 env[3074]: Started wazuh-syscheckd... Apr 29 10:21:41 ip-172-31-36-193 env[3074]: Started wazuh-logcollector... Apr 29 10:21:42 ip-172-31-36-193 env[3074]: Started wazuh-modulesd... Apr 29 10:21:44 ip-172-31-36-193 env[3074]: Completed. Apr 29 10:21:44 ip-172-31-36-193 systemd[1]: Started Wazuh agent. ```

Windows Agent

System information
```console C:\Users\Jenkins>systeminfo | findstr /B /C:"OS Name" /B /C:"OS Version" OS Name: Microsoft Windows 11 Pro OS Version: 10.0.22621 N/A Build 22621 ```
Agent version
```console PS C:\> cd 'C:\Program Files (x86)\ossec-agent\' PS C:\Program Files (x86)\ossec-agent> (Get-Command .\wazuh-agent.exe).FileVersionInfo ProductVersion FileVersion FileName -------------- ----------- -------- v4.8.0 v4.8.0 C:\Program Files (x86)\ossec-agent\wazuh-agent.exe ```
Agent status
```console C:\Windows\System32>NET START Wazuh The requested service has already been started. ```
santipadilla commented 2 weeks ago

Before testing

Both agents connected to the master
```console root@ip-172-31-40-132:/home/ubuntu# /var/ossec/bin/agent_control -l Wazuh agent_control. List of available agents: ID: 000, Name: ip-172-31-40-132 (server), IP: 127.0.0.1, Active/Local ID: 001, Name: ip-172-31-36-193, IP: any, Active ID: 002, Name: DESKTOP-AQ2R8SM, IP: any, Active ```
Total vulnerabilities
```console root@ip-172-31-40-132:/home/ubuntu# curl -k -u admin:0uWC?FglvfSLUCYFRF9.W42LBdBsnyR6 https://172.31.42.75:9200/wazuh-states-vulnerabilities-wazuh-node01/_count?pretty=true -H 'Content-Type: application/json' { "count" : 616, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 } } ```
Ubuntu agent vulnerabilities
```console root@ip-172-31-40-132:/home/ubuntu# curl -k -u admin:0uWC?FglvfSLUCYFRF9.W42LBdBsnyR6 https://172.31.42.75:9200/wazuh-states-vulnerabilities-wazuh-node01/_count?pretty=true -H 'Content-Type: application/json' -d '{ "query": { "term": { "agent.id": "001" } } }' { "count" : 98, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 } } ```
Windows agent vulnerabilities
```console root@ip-172-31-40-132:/home/ubuntu# curl -k -u admin:0uWC?FglvfSLUCYFRF9.W42LBdBsnyR6 https://172.31.42.75:9200/wazuh-states-vulnerabilities-wazuh-node01/_count?pretty=true -H 'Content-Type: application/json' -d '{ "query": { "term": { "agent.id": "002" } } }' { "count" : 452, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 } } ```
santipadilla commented 2 weeks ago

Case 1: Change of worker due to agent restart

Given a configured master And two workers configured And an agent connects to the master node And the agent with state Active When the agent is restarted and changes worker Then No doubling of the vulnerability index

santipadilla commented 2 weeks ago

Case 2: Change of worker due to connection loss

Given a configured master And two workers configured And an agent connects to the worker node And the agent with state Active When connection lost and worker changed Then No doubling of the vulnerability index

santipadilla commented 2 weeks ago

Case 3: Change of worker due to worker restart

Given a configured master And two workers configured And an agent connects to the worker node And the agent with state Active When Restart and change of worker Then No doubling of the vulnerability index

juliamagan commented 2 weeks ago

After a meeting with @davidjiglesias and @wazuh/devel-qa-div2, it has been decided that we will not continue with this issue since the development that was intended to be tested here will not be used in the end.