mhamra
commented
2 weeks ago UPDATE
The manager installation test calls the Utils.check_inventory, which returns False when the ssh authentication fails, but the method doesn't check the return value.
This value must be tested, or the Utils.check_inventory method must raise the exception.
Description
Running this workflow file with the
--threads 3workflow.yaml
``` version: 0.1 description: This workflow is used to test manager deployment for DDT1 PoC variables: manager-os: - linux-ubuntu-20.04-amd64 - linux-ubuntu-22.04-amd64 - linux-oracle-9-amd64 # - linux-amazon-2-amd64 # - linux-redhat-7-amd64 # - linux-redhat-8-amd64 # - linux-redhat-9-amd64 # - linux-centos-7-amd64 # - linux-centos-8-amd64 # - linux-debian-10-amd64 # - linux-debian-11-amd64 # - linux-debian-12-amd64 infra-provider: vagrant working-dir: /tmp/dtt1-poc tasks: # Unique manager allocate task - task: "allocate-manager-{manager}" description: "Allocate resources for the manager." do: this: process with: path: python3 args: - modules/allocation/main.py - action: create - provider: "{infra-provider}" - size: large - composite-name: "{manager}" - inventory-output: "{working-dir}/manager-{manager}/inventory.yaml" - track-output: "{working-dir}/manager-{manager}/track.yaml" on-error: "abort-all" foreach: - variable: manager-os as: manager # Generic manager test task - task: "run-manager-tests" description: "Run tests install for the manager." do: this: process with: path: python3 args: - modules/testing/main.py - targets: - wazuh-1: "{working-dir}/manager-linux-ubuntu-20.04-amd64/inventory.yaml" - wazuh-2: "{working-dir}/manager-linux-ubuntu-22.04-amd64/inventory.yaml" - wazuh-3: "{working-dir}/manager-linux-oracle-9-amd64/inventory.yaml" # - wazuh-4: "{working-dir}/manager-linux-centos-7-amd64/inventory.yaml" # - wazuh-5: "{working-dir}/manager-linux-amazon-2-amd64/inventory.yaml" # - wazuh-6: "{working-dir}/manager-linux-redhat-7-amd64/inventory.yaml" # - wazuh-7: "{working-dir}/manager-linux-redhat-8-amd64/inventory.yaml" # - wazuh-8: "{working-dir}/manager-linux-redhat-9-amd64/inventory.yaml" # - wazuh-9: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml" # - wazuh-10: "{working-dir}/mmarcelo@marcelo-B460-AORUS-PRO-AC:~/wazuh/wazuh-qa$ tail -f /tmp/workflow.log [2024-04-17 12:15:59,960] [INFO] [744222] [MainThread] [workflow_engine]: Executing DAG tasks. [2024-04-17 12:15:59,960] [INFO] [744222] [MainThread] [workflow_engine]: Executing tasks in parallel. [2024-04-17 12:15:59,960] [INFO] [744222] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Starting task. [2024-04-17 12:15:59,961] [INFO] [744222] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task. [2024-04-17 12:15:59,961] [INFO] [744222] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Starting task. [2024-04-17 12:16:00,188] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa [2024-04-17 12:16:00,189] [DEBUG] ALLOCATOR: No config provided. Generating from payload [2024-04-17 12:16:00,189] [DEBUG] ALLOCATOR: Generating new key pair [2024-04-17 12:16:00,192] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa [2024-04-17 12:16:00,193] [DEBUG] ALLOCATOR: No config provided. Generating from payload [2024-04-17 12:16:00,193] [DEBUG] ALLOCATOR: Generating new key pair [2024-04-17 12:16:00,214] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa [2024-04-17 12:16:00,214] [DEBUG] ALLOCATOR: No config provided. Generating from payload [2024-04-17 12:16:00,214] [DEBUG] ALLOCATOR: Generating new key pair [2024-04-17 12:16:03,600] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance. [2024-04-17 12:16:03,600] [INFO] ALLOCATOR: Instance VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3 created. [2024-04-17 12:16:03,601] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance. [2024-04-17 12:16:03,602] [INFO] ALLOCATOR: Instance VAGRANT-2F9CB804-CC23-4972-BADF-4DC8A2403068 created. [2024-04-17 12:16:03,902] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance. [2024-04-17 12:16:03,902] [INFO] ALLOCATOR: Instance VAGRANT-2504FAA1-F821-46EE-B8A9-86E283308F39 created. [2024-04-17 12:16:59,745] [INFO] ALLOCATOR: Instance VAGRANT-2504FAA1-F821-46EE-B8A9-86E283308F39 started. [2024-04-17 12:17:02,008] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml [2024-04-17 12:17:02,887] [INFO] ALLOCATOR: Instance VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3 started. [2024-04-17 12:17:04,281] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/track.yaml [2024-04-17 12:17:04,330] [INFO] [744222] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 64.37 seconds. [2024-04-17 12:17:04,386] [INFO] ALLOCATOR: Instance VAGRANT-2F9CB804-CC23-4972-BADF-4DC8A2403068 started. [2024-04-17 12:17:05,148] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml [2024-04-17 12:17:06,746] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml [2024-04-17 12:17:07,390] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/track.yaml [2024-04-17 12:17:07,446] [INFO] [744222] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Finished task in 67.49 seconds. [2024-04-17 12:17:09,054] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-oracle-9-amd64/track.yaml [2024-04-17 12:17:09,101] [INFO] [744222] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Finished task in 69.13 seconds. [2024-04-17 12:17:09,116] [INFO] [744222] [ThreadPoolExecutor-0_1] [workflow_engine]: [run-manager-tests] Starting task. [2024-04-17 12:17:09,374] [INFO] TESTER: Running tests for 192.168.57.2 [2024-04-17 12:17:09,375] [INFO] TESTER: Running tests for 192.168.57.2 [2024-04-17 12:17:09,375] [INFO] TESTER: Running tests for 192.168.57.2 [2024-04-17 12:17:09,375] [DEBUG] TESTER: Using extra vars: {'component': 'manager', 'wazuh_version': '4.7.3', 'wazuh_revision': '40714', 'wazuh_branch': None, 'working_dir': '/tmp/tests', 'live': True, 'hosts_ip': ['192.168.57.2', '192.168.57.2', '192.168.57.2'], 'targets': '{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml, wazuh-2: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, wazuh-3: /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml}', 'dependencies': '{}', 'local_host_path': '/home/marcelo/wazuh/wazuh-qa/deployability', 'current_user': 'marcelo'} [2024-04-17 12:17:09,377] [DEBUG] TESTER: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/setup.yml [2024-04-17 12:17:09,385] [DEBUG] TESTER: Using inventory: {'all': {'hosts': {'192.168.57.2': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3/instance_key'}}}} [2024-04-17 12:17:09,385] [DEBUG] TESTER: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['192.168.57.2', '192.168.57.2', '192.168.57.2']}]}] [2024-04-17 12:17:10,984] [DEBUG] TESTER: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['192.168.57.2', '192.168.57.2', '192.168.57.2']}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}} [2024-04-17 12:17:10,986] [DEBUG] TESTER: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml [2024-04-17 12:17:10,987] [DEBUG] TESTER: Using inventory: {'all': {'hosts': {'192.168.57.2': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3/instance_key'}}}} [2024-04-17 12:17:10,987] [DEBUG] TESTER: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test install for manager', 'command': "python3 -m pytest modules/testing/tests/test_manager/test_install.py -v --wazuh_version=4.7.3 --wazuh_revision=40714 --component=manager --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml, wazuh-2: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, wazuh-3: /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] [2024-04-17 12:17:12,591] [INFO] TESTER: Checking connection to ubuntu-20.04 [2024-04-17 12:17:12,853] [ERROR] TESTER: Authentication error. Check SSH credentials in ubuntu-20.04 anager-linux-debian-10-amd64/inventory.yaml" # - wazuh-11: "{working-dir}/manager-linux-debian-11-amd64/inventory.yaml" # - wazuh-12: "{working-dir}/manager-linux-debian-12-amd64/inventory.yaml" - tests: "install,restart,stop,uninstall" - component: "manager" - wazuh-version: "4.7.3" - wazuh-revision: "40714" - live: "True" depends-on: - "allocate-manager-linux-ubuntu-20.04-amd64" - "allocate-manager-linux-ubuntu-22.04-amd64" - "allocate-manager-linux-oracle-9-amd64" ```the testing module logs that the instances have the same IP address, which causes the
sshlogin to fail.workflow.log
``` marcelo@marcelo-B460-AORUS-PRO-AC:~/wazuh/wazuh-qa$ tail -f /tmp/workflow.log [2024-04-17 12:15:59,960] [INFO] [744222] [MainThread] [workflow_engine]: Executing DAG tasks. [2024-04-17 12:15:59,960] [INFO] [744222] [MainThread] [workflow_engine]: Executing tasks in parallel. [2024-04-17 12:15:59,960] [INFO] [744222] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Starting task. [2024-04-17 12:15:59,961] [INFO] [744222] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Starting task. [2024-04-17 12:15:59,961] [INFO] [744222] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Starting task. [2024-04-17 12:16:00,188] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa [2024-04-17 12:16:00,189] [DEBUG] ALLOCATOR: No config provided. Generating from payload [2024-04-17 12:16:00,189] [DEBUG] ALLOCATOR: Generating new key pair [2024-04-17 12:16:00,192] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa [2024-04-17 12:16:00,193] [DEBUG] ALLOCATOR: No config provided. Generating from payload [2024-04-17 12:16:00,193] [DEBUG] ALLOCATOR: Generating new key pair [2024-04-17 12:16:00,214] [INFO] ALLOCATOR: Creating instance at /tmp/wazuh-qa [2024-04-17 12:16:00,214] [DEBUG] ALLOCATOR: No config provided. Generating from payload [2024-04-17 12:16:00,214] [DEBUG] ALLOCATOR: Generating new key pair [2024-04-17 12:16:03,600] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance. [2024-04-17 12:16:03,600] [INFO] ALLOCATOR: Instance VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3 created. [2024-04-17 12:16:03,601] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance. [2024-04-17 12:16:03,602] [INFO] ALLOCATOR: Instance VAGRANT-2F9CB804-CC23-4972-BADF-4DC8A2403068 created. [2024-04-17 12:16:03,902] [DEBUG] ALLOCATOR: Vagrantfile created. Creating instance. [2024-04-17 12:16:03,902] [INFO] ALLOCATOR: Instance VAGRANT-2504FAA1-F821-46EE-B8A9-86E283308F39 created. [2024-04-17 12:16:59,745] [INFO] ALLOCATOR: Instance VAGRANT-2504FAA1-F821-46EE-B8A9-86E283308F39 started. [2024-04-17 12:17:02,008] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml [2024-04-17 12:17:02,887] [INFO] ALLOCATOR: Instance VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3 started. [2024-04-17 12:17:04,281] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/track.yaml [2024-04-17 12:17:04,330] [INFO] [744222] [ThreadPoolExecutor-0_1] [workflow_engine]: [allocate-manager-linux-ubuntu-22.04-amd64] Finished task in 64.37 seconds. [2024-04-17 12:17:04,386] [INFO] ALLOCATOR: Instance VAGRANT-2F9CB804-CC23-4972-BADF-4DC8A2403068 started. [2024-04-17 12:17:05,148] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml [2024-04-17 12:17:06,746] [INFO] ALLOCATOR: Inventory file generated at /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml [2024-04-17 12:17:07,390] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/track.yaml [2024-04-17 12:17:07,446] [INFO] [744222] [ThreadPoolExecutor-0_0] [workflow_engine]: [allocate-manager-linux-ubuntu-20.04-amd64] Finished task in 67.49 seconds. [2024-04-17 12:17:09,054] [INFO] ALLOCATOR: Track file generated at /tmp/dtt1-poc/manager-linux-oracle-9-amd64/track.yaml [2024-04-17 12:17:09,101] [INFO] [744222] [ThreadPoolExecutor-0_2] [workflow_engine]: [allocate-manager-linux-oracle-9-amd64] Finished task in 69.13 seconds. [2024-04-17 12:17:09,116] [INFO] [744222] [ThreadPoolExecutor-0_1] [workflow_engine]: [run-manager-tests] Starting task. [2024-04-17 12:17:09,374] [INFO] TESTER: Running tests for 192.168.57.2 [2024-04-17 12:17:09,375] [INFO] TESTER: Running tests for 192.168.57.2 [2024-04-17 12:17:09,375] [INFO] TESTER: Running tests for 192.168.57.2 [2024-04-17 12:17:09,375] [DEBUG] TESTER: Using extra vars: {'component': 'manager', 'wazuh_version': '4.7.3', 'wazuh_revision': '40714', 'wazuh_branch': None, 'working_dir': '/tmp/tests', 'live': True, 'hosts_ip': ['192.168.57.2', '192.168.57.2', '192.168.57.2'], 'targets': '{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml, wazuh-2: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, wazuh-3: /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml}', 'dependencies': '{}', 'local_host_path': '/home/marcelo/wazuh/wazuh-qa/deployability', 'current_user': 'marcelo'} [2024-04-17 12:17:09,377] [DEBUG] TESTER: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/setup.yml [2024-04-17 12:17:09,385] [DEBUG] TESTER: Using inventory: {'all': {'hosts': {'192.168.57.2': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3/instance_key'}}}} [2024-04-17 12:17:09,385] [DEBUG] TESTER: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['192.168.57.2', '192.168.57.2', '192.168.57.2']}]}] [2024-04-17 12:17:10,984] [DEBUG] TESTER: Playbook [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Cleaning old key ssh-keygen registries', 'ansible.builtin.command': {'cmd': "ssh-keygen -f /home/marcelo/.ssh/known_hosts -R ''"}, 'loop': ['192.168.57.2', '192.168.57.2', '192.168.57.2']}]}] finished with status {'skipped': {}, 'ok': {'localhost': 2}, 'dark': {}, 'failures': {}, 'ignored': {}, 'rescued': {}, 'processed': {'localhost': 1}, 'changed': {'localhost': 1}} [2024-04-17 12:17:10,986] [DEBUG] TESTER: Rendering template /home/marcelo/wazuh/wazuh-qa/deployability/modules/testing/playbooks/test.yml [2024-04-17 12:17:10,987] [DEBUG] TESTER: Using inventory: {'all': {'hosts': {'192.168.57.2': {'ansible_port': 22, 'ansible_user': 'vagrant', 'ansible_ssh_private_key_file': '/tmp/wazuh-qa/VAGRANT-F72E5B3B-11D7-46DE-A8B3-8557E9D5F6F3/instance_key'}}}} [2024-04-17 12:17:10,987] [DEBUG] TESTER: Running playbook: [{'hosts': 'localhost', 'become': True, 'become_user': 'marcelo', 'tasks': [{'name': 'Test install for manager', 'command': "python3 -m pytest modules/testing/tests/test_manager/test_install.py -v --wazuh_version=4.7.3 --wazuh_revision=40714 --component=manager --dependencies='{}' --targets='{wazuh-1: /tmp/dtt1-poc/manager-linux-ubuntu-20.04-amd64/inventory.yaml, wazuh-2: /tmp/dtt1-poc/manager-linux-ubuntu-22.04-amd64/inventory.yaml, wazuh-3: /tmp/dtt1-poc/manager-linux-oracle-9-amd64/inventory.yaml}' --live=True -s", 'args': {'chdir': '/home/marcelo/wazuh/wazuh-qa/deployability'}}]}] [2024-04-17 12:17:12,591] [INFO] TESTER: Checking connection to ubuntu-20.04 [2024-04-17 12:17:12,853] [ERROR] TESTER: Authentication error. Check SSH credentials in ubuntu-20.04 ```The workflow execution stopped at that point without generating an exception or exiting by a timeout.