search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
64 stars 30 forks source link

Validate the behavior in wazuh-db during agent-groups insertion #3868

Closed QU3B1M closed 1 year ago

QU3B1M commented 1 year ago
Target version Related issue Related PR
4.4.0 #15966 #16049

Description

This issue aims to validate the Inconsistent behavior in wazuh-db during agent-groups insertion is fixed. The master and the workers must always have the groups-hash updated and synchronized between them

Proposed checks

Environment

Configuration
wazuh installation multi-node cluster
nodes master + 2 workers

Fix application methods

OS

Configuration and considerations

QU3B1M commented 1 year ago

Review data

Tester PR commit
@QU3B1M cd53064

Testing environment

OS OS version Deployment Image/AMI Notes
Ubuntu 22.04 LOCAL \| Vagrant generic/ubuntu2204

Tested packages

wazuh-manager
deb

Status

Conclusion :green_circle:

Test execution was successful :green_circle:. When this manual test is approved and the core's PR merged, this QA PR that fixes wazuh_db automated tests must also be merged.

QU3B1M commented 1 year ago

Error reproduction

  1. Given a cluster with 2 worker nodes
  2. And 50k wazuh-agents connected assigned to 2 groups
  3. When delete all the wazuh-agents
  4. And immediately restart the master node.

  5. Then the groups_hash reported from all the cluster nodes are correct :red_circle:.

    group hash origin value
    Real group hash da39a3ee5e6b4b0d3255bfef95601890afd80709
    Reported by master 4672826f83e477ad4e24bbe43a718a5fa89e9c24
    Reported by worker-1 94d6180a1569171c230635d7ccdc8fa488bf9e93
    Reported by worker-2 94d6180a1569171c230635d7ccdc8fa488bf9e93
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4672826f83e477ad4e24bbe43a718a5fa89e9c24"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"94d6180a1569171c230635d7ccdc8fa488bf9e93"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"94d6180a1569171c230635d7ccdc8fa488bf9e93"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: da39a3ee5e6b4b0d3255bfef95601890afd80709 ```
QU3B1M commented 1 year ago

Error in global_hash

A bug was found in the generation of global_hash where it includes the group_hash of the agent 000, as it is typically Null in most cases it does not have any impact, but if for any reason the group_hash of the agent 000 is not Null it will causes inconsistencies in the global_group_hash reported from the master and the workers. Testing paused until it is fixed.

QU3B1M commented 1 year ago

Bug fixed - Resume testing

The error in the global_hash generation was fixed in this PR, the testing is restarted as there is a new commit, everything should be tested again.

After this fix one of our IT tests started to fail, after some research it was found that the test test_wazuh_db/test_sync_agent_groups_get.py compares the global_hash generated by wazuh to a hardcoded hash from the test case yaml. An issue was created to fix this test:

QU3B1M commented 1 year ago

Testing results :green_circle:

Unfixed

Error reproduction > In this case the hashes are not null because wazuh is restarted before it finishes removing all the agents, its ideal to test the `global_group_hash` is the same in each node. 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 50k `wazuh-agents` connected assigned to 2 groups. 3. **When** delete all the `wazuh-agents`. 4. **And** immediately restart the `master` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are not correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `da39a3ee5e6b4b0d3255bfef95601890afd80709`| | Reported by master | `4672826f83e477ad4e24bbe43a718a5fa89e9c24`| | Reported by worker-1| `94d6180a1569171c230635d7ccdc8fa488bf9e93`| | Reported by worker-2| `94d6180a1569171c230635d7ccdc8fa488bf9e93`|
full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4672826f83e477ad4e24bbe43a718a5fa89e9c24"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"94d6180a1569171c230635d7ccdc8fa488bf9e93"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"94d6180a1569171c230635d7ccdc8fa488bf9e93"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: da39a3ee5e6b4b0d3255bfef95601890afd80709 ```

Upgrade

Upgrade process - Upgrade `master-node` ```console dpkg -i wazuh-manager_4.4.0-0.commitcd53064_amd64.deb dpkg: warning: downgrading wazuh-manager from 4.4.0-1 to 4.4.0-0.commitcd53064 (Reading database ... 101127 files and directories currently installed.) Preparing to unpack wazuh-manager_4.4.0-0.commitcd53064_amd64.deb ... Unpacking wazuh-manager (4.4.0-0.commitcd53064) over (4.4.0-1) ... Setting up wazuh-manager (4.4.0-0.commitcd53064) ... ``` - Upgrade `worker-1-node` ```console dpkg -i wazuh-manager_4.4.0-0.commitcd53064_amd64.deb dpkg: warning: downgrading wazuh-manager from 4.4.0-1 to 4.4.0-0.commitcd53064 (Reading database ... 101127 files and directories currently installed.) Preparing to unpack wazuh-manager_4.4.0-0.commitcd53064_amd64.deb ... Unpacking wazuh-manager (4.4.0-0.commitcd53064) over (4.4.0-1) ... Setting up wazuh-manager (4.4.0-0.commitcd53064) ... ``` - Upgrade `worker-2-node` ```console dpkg -i wazuh-manager_4.4.0-0.commitcd53064_amd64.deb dpkg: warning: downgrading wazuh-manager from 4.4.0-1 to 4.4.0-0.commitcd53064 (Reading database ... 101127 files and directories currently installed.) Preparing to unpack wazuh-manager_4.4.0-0.commitcd53064_amd64.deb ... Unpacking wazuh-manager (4.4.0-0.commitcd53064) over (4.4.0-1) ... Setting up wazuh-manager (4.4.0-0.commitcd53064) ... ```
group_hash is generated correctly when new agents are connected :green_circle: -
Connect one agent :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect one `wazuh-agent`. 3. **And** assign it to two groups. 4. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`| | Reported by master | `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`| | Reported by worker-1| `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`| | Reported by worker-2| `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`|
full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"e86f56cf2a0cd660c04ecc5d83992edf87bc6608"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"e86f56cf2a0cd660c04ecc5d83992edf87bc6608"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"e86f56cf2a0cd660c04ecc5d83992edf87bc6608"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: e86f56cf2a0cd660c04ecc5d83992edf87bc6608 ```
  • Connect one agent to each node :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect one `wazuh-agent` to each `node`. 3. **And** assign them to two groups. 4. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `4e28b9e834a776cf1626288cc473b5efd1db96c5`| | Reported by master | `4e28b9e834a776cf1626288cc473b5efd1db96c5`| | Reported by worker-1| `4e28b9e834a776cf1626288cc473b5efd1db96c5`| | Reported by worker-2| `4e28b9e834a776cf1626288cc473b5efd1db96c5`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4e28b9e834a776cf1626288cc473b5efd1db96c5"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4e28b9e834a776cf1626288cc473b5efd1db96c5"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4e28b9e834a776cf1626288cc473b5efd1db96c5"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 4e28b9e834a776cf1626288cc473b5efd1db96c5 ```
  • Connect 50k agents and restart the master :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect 50k `wazuh-agents`. 3. **And** assign them to two groups. 4. **And** inmediately restart the `master` 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `d157d68b38e320bb75e139475746c23cef968510`| | Reported by master | `d157d68b38e320bb75e139475746c23cef968510`| | Reported by worker-1| `d157d68b38e320bb75e139475746c23cef968510`| | Reported by worker-2| `d157d68b38e320bb75e139475746c23cef968510`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"d157d68b38e320bb75e139475746c23cef968510"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"d157d68b38e320bb75e139475746c23cef968510"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"d157d68b38e320bb75e139475746c23cef968510"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: d157d68b38e320bb75e139475746c23cef968510 ```
  • Connect 10k agents to each node and restart the master :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect 10k `wazuh-agents` to each `node`. 3. **And** assign them to two groups. 4. **And** inmediately restart the `master` 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by master | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-1| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-2| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 3ba57aac7e8e364583fd46f900c9714ccc8fffc8 ```
  • Connect 10k agents to each node and restart the workers :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect 10k `wazuh-agents` to each `node`. 3. **And** assign them to two groups. 4. **And** inmediately restart the `master` 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by master | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-1| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-2| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 3ba57aac7e8e364583fd46f900c9714ccc8fffc8 ```

group_hash is generated correctly when agents are removed :green_circle: -
Remove 50k agents and restart the master :green_circle: > In this case the hashes are not null because wazuh is restarted before it finishes removing all the agents (just like in the [error reproduction](https://github.com/wazuh/wazuh-qa/issues/3868#issuecomment-1414100676)) and its ideal to test the `global_group_hash`is the same in each node. 1. **Given** a `cluster` with 2 `worker` nodes 2. **And** 50k `wazuh-agents` connected assigned to 2 groups 3. **When** delete all the `wazuh-agents`. 4. **And** immediately restart the `master` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `ee04b307180650e553422cdae8e879794d76423b`| | Reported by master | `ee04b307180650e553422cdae8e879794d76423b`| | Reported by worker-1| `ee04b307180650e553422cdae8e879794d76423b`| | Reported by worker-2| `ee04b307180650e553422cdae8e879794d76423b`|
full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"ee04b307180650e553422cdae8e879794d76423b"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"ee04b307180650e553422cdae8e879794d76423b"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"ee04b307180650e553422cdae8e879794d76423b"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: ee04b307180650e553422cdae8e879794d76423b ```
  • Remove 10k agents from each node and restart the master :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 10k `wazuh-agents` connected to each node (30k total) assigned to 2 groups. 3. **When** delete all the `wazuh-agents`. 4. **And** immediately restart the `master` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `null`| | Reported by master | `null`| | Reported by worker-1| `null`| | Reported by worker-2| `null`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: None ```
  • Remove 10k agents from each node and restart the workers :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 10k `wazuh-agents` connected to each node (30k total) assigned to 2 groups. 3. **When** delete all the `wazuh-agents`. 4. **And** immediately restart both `workers` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `null`| | Reported by master | `null`| | Reported by worker-1| `null`| | Reported by worker-2| `null`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: None ```
  • Remove one agent :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 10k `wazuh-agents` connected assigned to 2 groups. 3. **When** delete one `wazuh-agents`. 4. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `3f127cd2cd733eec8354f4147d3a2a534dad729c`| | Reported by master | `3f127cd2cd733eec8354f4147d3a2a534dad729c`| | Reported by worker-1| `3f127cd2cd733eec8354f4147d3a2a534dad729c`| | Reported by worker-2| `3f127cd2cd733eec8354f4147d3a2a534dad729c`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3f127cd2cd733eec8354f4147d3a2a534dad729c"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3f127cd2cd733eec8354f4147d3a2a534dad729c"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3f127cd2cd733eec8354f4147d3a2a534dad729c"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 3f127cd2cd733eec8354f4147d3a2a534dad729c ```

Fresh Install

group_hash is generated correctly when new agents are connected :green_circle: -
Connect one agent :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect one `wazuh-agent`. 3. **And** assign it to two groups. 4. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`| | Reported by master | `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`| | Reported by worker-1| `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`| | Reported by worker-2| `e86f56cf2a0cd660c04ecc5d83992edf87bc6608`|
full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"e86f56cf2a0cd660c04ecc5d83992edf87bc6608"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"e86f56cf2a0cd660c04ecc5d83992edf87bc6608"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"e86f56cf2a0cd660c04ecc5d83992edf87bc6608"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: e86f56cf2a0cd660c04ecc5d83992edf87bc6608 ```
  • Connect one agent to each node :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect one `wazuh-agent` to each `node`. 3. **And** assign them to two groups. 4. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `4e28b9e834a776cf1626288cc473b5efd1db96c5`| | Reported by master | `4e28b9e834a776cf1626288cc473b5efd1db96c5`| | Reported by worker-1| `4e28b9e834a776cf1626288cc473b5efd1db96c5`| | Reported by worker-2| `4e28b9e834a776cf1626288cc473b5efd1db96c5`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4e28b9e834a776cf1626288cc473b5efd1db96c5"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4e28b9e834a776cf1626288cc473b5efd1db96c5"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"4e28b9e834a776cf1626288cc473b5efd1db96c5"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 4e28b9e834a776cf1626288cc473b5efd1db96c5 ```
  • Connect 50k agents and restart the master :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect 50k `wazuh-agents`. 3. **And** assign them to two groups. 4. **And** inmediately restart the `master` 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `d157d68b38e320bb75e139475746c23cef968510`| | Reported by master | `d157d68b38e320bb75e139475746c23cef968510`| | Reported by worker-1| `d157d68b38e320bb75e139475746c23cef968510`| | Reported by worker-2| `d157d68b38e320bb75e139475746c23cef968510`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"d157d68b38e320bb75e139475746c23cef968510"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"d157d68b38e320bb75e139475746c23cef968510"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"d157d68b38e320bb75e139475746c23cef968510"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: d157d68b38e320bb75e139475746c23cef968510 ```
  • Connect 10k agents to each node and restart the master :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect 10k `wazuh-agents` to each `node`. 3. **And** assign them to two groups. 4. **And** inmediately restart the `master` 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by master | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-1| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-2| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 3ba57aac7e8e364583fd46f900c9714ccc8fffc8 ```
  • Connect 10k agents to each node and restart the workers :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **When** connect 10k `wazuh-agents` to each `node`. 3. **And** assign them to two groups. 4. **And** inmediately restart the `master` 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by master | `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-1| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`| | Reported by worker-2| `3ba57aac7e8e364583fd46f900c9714ccc8fffc8`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3ba57aac7e8e364583fd46f900c9714ccc8fffc8"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 3ba57aac7e8e364583fd46f900c9714ccc8fffc8 ```

group_hash is generated correctly when agents are removed :green_circle: -
Remove 50k agents and restart the master :green_circle: > In this case the hashes are not null because wazuh is restarted before it finishes removing all the agents (just like in the [error reproduction](https://github.com/wazuh/wazuh-qa/issues/3868#issuecomment-1414100676)) and its ideal to test the `global_group_hash`is the same in each node. 1. **Given** a `cluster` with 2 `worker` nodes 2. **And** 50k `wazuh-agents` connected assigned to 2 groups 3. **When** delete all the `wazuh-agents` 4. **And** immediately restart the `master` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `1a89ed3e4021283ffb7a24ece36fbe108f395bdf`| | Reported by master | `1a89ed3e4021283ffb7a24ece36fbe108f395bdf`| | Reported by worker-1| `1a89ed3e4021283ffb7a24ece36fbe108f395bdf`| | Reported by worker-2| `1a89ed3e4021283ffb7a24ece36fbe108f395bdf`|
full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"1a89ed3e4021283ffb7a24ece36fbe108f395bdf"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"1a89ed3e4021283ffb7a24ece36fbe108f395bdf"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"1a89ed3e4021283ffb7a24ece36fbe108f395bdf"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 1a89ed3e4021283ffb7a24ece36fbe108f395bdf ```
  • Remove 10k agents from each node and restart the master :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 10k `wazuh-agents` connected to each node (30k total) assigned to 2 groups. 3. **When** delete all the `wazuh-agents` 4. **And** immediately restart the `master` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `null`| | Reported by master | `null`| | Reported by worker-1| `null`| | Reported by worker-2| `null`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: None ```
  • Remove 10k agents from each node and restart the workers :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 10k `wazuh-agents` connected to each node (30k total) assigned to 2 groups. 3. **When** delete all the `wazuh-agents`. 4. **And** immediately restart both `workers` node. 5. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `null`| | Reported by master | `null`| | Reported by worker-1| `null`| | Reported by worker-2| `null`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":null}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: None ```
  • Remove one agent :green_circle: 1. **Given** a `cluster` with 2 `worker` nodes. 2. **And** 10k `wazuh-agents` connected assigned to 2 groups. 3. **When** delete one `wazuh-agents`. 4. **Then** the `groups_hash` reported from all the `cluster nodes` are correct. | group hash origin | value | |---------------------|-------------------------------------------| | Real group hash | `3f127cd2cd733eec8354f4147d3a2a534dad729c`| | Reported by master | `3f127cd2cd733eec8354f4147d3a2a534dad729c`| | Reported by worker-1| `3f127cd2cd733eec8354f4147d3a2a534dad729c`| | Reported by worker-2| `3f127cd2cd733eec8354f4147d3a2a534dad729c`|
    full logs `reported hash by master` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3f127cd2cd733eec8354f4147d3a2a534dad729c"}]'] ``` `reported hash by worker-1` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3f127cd2cd733eec8354f4147d3a2a534dad729c"}]'] ``` `reported hash by worker-2` ```powershell python3 send_msg.py 'global sync-agent-groups-get {"condition": "sync_status", "set_synced": false, "get_global_hash": true}' ['ok', '[{"data":[],"hash":"3f127cd2cd733eec8354f4147d3a2a534dad729c"}]'] ``` `Real group hash` ```powershell python3 calculate_group_hash.py Global groups hash: 3f127cd2cd733eec8354f4147d3a2a534dad729c ```

jmv74211 commented 1 year ago

Closing conclusion 👍🏼

Everything seems to be working properly.