Closed verdx closed 5 months ago
manager_test
: When manager_test
is called, the tests are run correctly both with Ubuntu and CentOS managers. Package 4.3.0-1 fails on some of the tests but for 4.4.4-1, they are all passed.agent_test
:
master
and package 4.3.0-1
it fails on one of the FIM tests and for branch 4.4
and package 4.4.4-1
it fails with an error with Windows, but all tests are passed correctly. The error in Windows appears in the task Fetch zip HTML for Windows
and is explained in the general commentary.
agent_windows_html_report_Test_integration_B40405_20230620161732.zip agent_ubuntu_html_report_Test_integration_B40405_20230620161732.zip agent_solaris_html_report_Test_integration_B40405_20230620161732.zip agent_macos_html_report_Test_integration_B40405_20230620161732.zip agent_centos_html_report_Test_integration_B40405_20230620161732.zip
On an agent_test
, launched with a CentOS manager and using Jenkins branch 4.4, the tests have run correctly except for an error in task Fetch zip HTML report from Windows
.
manager_test
: When manager_test
is called, the tests are run and passed correctly both with Ubuntu and CentOS managers.agent_test
:
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
:green_circle: manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all tests
agent_test
:
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
It is similar to the error seen in test_fim
, but this time not only on Windows
manager_test
:
Fetch zip HTML report
manager_html_report_Test_integration_B40449_20230622152233.zipagent_test
:
It is similar to the error seen in test_fim
, but this time not only on Windows
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
manager_test
:
Fetch zip HTML report
agent_test
:
Fetch zip HTML report
:red_circle: manager_test
: Using both CentOS and Ubuntu managers, the manager test fails with an error in Fetch zip HTML report
. The test is not passed
[Uploading manager_html_report_Test_integration_B40460_20230622165420.zip…]()
agent_test
:
:red_circle: Using a CentOS manager the test is correctly run but fails to pass for Solaris. agent_solaris_html_report_Test_integration_B40458_20230622165334.zip
:yellow_circle: Using an Ubuntu manager, it fails for the Linux agents trying to mount the NFS. It finishes UNSTABLE and the errors are furtherly explained here
:red_circle: manager_test
: Using both CentOS and Ubuntu managers, the manager test fails with the same error as the agent_test
with the CentOS manager.
agent_test
:
:red_circle: Using a CentOS manager the test fails with the seen error of Fetch zip HTML report
and a different error when calling pytest
.
:yellow_circle: Using an Ubuntu manager, it fails for the Linux agents trying to mount the NFS. It finishes UNSTABLE and the errors are furtherly explained here
agent_test
manager_test
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
WINRM SEND INPUT
and Fetch zip HTML report
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly but the tests are not passedagent_test
:
agent_test
with CentOS ManagerThe agent reports are the following:
manager_test
both managersmanager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
Fetch zip HTML report
for all agents and the test fails tooagent_test
with CentOS managerAll tests fail in https://ci.wazuh.info/job/Test_integration/40483/
The agent reports are the following:
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
Fetch zip HTML report on Windows
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
Fetch zip HTML report on Windows
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly and passes all testsagent_test
:
Fetch zip HTML report on Windows
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly but some tests are not passed: test_validate_xml_feed_content and test_providers_missing_osmanager_test
with both Ubuntu and CentOSThe following tests are not passed for either manager system:
For the CentOS manager, a further test failed:
manager_test
: Using both CentOS and Ubuntu managers, the manager test runs correctly(with a Fetch zip HTML report
error) but the tests are not passedagent_test
:
Fetch zip HTML report on Windows
. The rest of the tests have run and passedmanager_tests
for both CentOS and Ubuntu systemsThe following test fails for both systems: test_wazuh_db/test_wazuhdb_getconfig.py::test_sync_agent_groups[/tmp/Test: Get internal config]
agent_test
with CentOS managerThe tests for agent_test
in Windows using a CentOS manager have failed without an explicit error and with no message. The rest of the tests have run and passed.
agent_windows_html_report_Test_integration_B40506_20230623165552.zip
manager_test
: The manager test fails when running using both CentOS and Ubuntu managers.agent_test
:
agent_test
with CentOS managerThe Linux pytests fail when the pipeline is called for agent_test
with a CentOS manager. The report is the following:
manager_html_report_Test_integration_B40512_20230623165949.zip
agent_test
with Ubuntu managerThe Linux pytests fail when the pipeline is called for agent_test
with a CentOS manager with a different error than using the CentOS manager. The report is the following:
manager_html_report_Test_integration_B40513_20230623170013.zip
agent_test
with CentOS managerAll the agents tests have failed with the same three errors
LGTM!
LGTM
Description
This issue aims to analyze and document all about the Integration test. Jenkins parameters, pytest location, steps in the test and current state of it.
Part of this investigation, although with manual testing, has been done on the issue https://github.com/wazuh/wazuh/issues/17007 and expressed on this spreadsheet https://docs.google.com/spreadsheets/d/1OL8QL_gWimxXNDzzAEhWyMrAfWegtZbBpRzPDEK54iQ/edit#gid=0
State
manager_test
: Whenmanager_test
is called, the tests are run correctly both with Ubuntu and CentOS managers.agent_test
:agent_tests
, if the Solaris agent is launched, it has sometimes failed with the error explained belowParameters
Specific parameters:
MANAGER_INSTANCE: [centos, ubuntu]
TARGET_TEST: [manager_test, agent_test]
AGENT_INSTANCES: only if agent_test, [centos, ubuntu, solaris, windows, macos]
TIER: only if manager_test, [0-6]
Tiers Tier 0test_basic_usage Verify if syscheck correctly detects basic behaviors, such as creating/modifying/deleting files in different FIM modes.
test_scan Check that scans occur at specific and established times.
Tier 1
Tier 2
Tier 3
Tier 4
Tier 5
Tier 6
-v --tb=short
Possible subtests
The whole test directory structure is explained here. Basically, the possible tests to run are the directories inside the directory
tests/integration
in https://github.com/wazuh/wazuh-qa. The test you want to run is defined in parameterTEST_PATH
. They are the following:All possible tests
Errors and problems
Solaris deployment on agent_test
The test was tried and when called with TARGET_TEST as
agent_test
, it fails on the Solaris deployment with the following error:Error in the Solaris deployment for the agent_test
``` TASK [Vagrant up] ************************************************************** 12:00:13 task path: /home/ec2-user/workspace/Test_integration/quality/deployments/ansible/create_vagrant.yaml:10 12:00:13 fatal: [QA_agent_solaris11_Test_integration_B40376_20230620104433]: FAILED! => { 12:00:13 "changed": true, 12:00:13 "cmd": [ 12:00:13 "vagrant", 12:00:13 "--name=QA_agent_solaris11_Test_integration_B40376_20230620104433", 12:00:13 "--copy-ssh-files", 12:00:13 "--bind-port=38768", 12:00:13 "--bind-ip=10.10.0.251", 12:00:13 "up", 12:00:13 "solaris-11" 12:00:13 ], 12:00:13 "delta": "0:01:06.680061", 12:00:13 "end": "2023-06-20 12:00:13.778560", 12:00:13 "invocation": { 12:00:13 "module_args": { 12:00:13 "_raw_params": "vagrant --name='QA_agent_solaris11_Test_integration_B40376_20230620104433' --copy-ssh-files --bind-port='38768' --bind-ip='10.10.0.251' up solaris-11", 12:00:13 "_uses_shell": false, 12:00:13 "argv": null, 12:00:13 "chdir": "/tmp/Test_integration_B40376_20230620104433/solaris_vagrant", 12:00:13 "creates": null, 12:00:13 "executable": null, 12:00:13 "removes": null, 12:00:13 "stdin": null, 12:00:13 "stdin_add_newline": true, 12:00:13 "strip_empty_ends": true, 12:00:13 "warn": true 12:00:13 } 12:00:13 }, 12:00:13 "rc": 1, 12:00:13 "start": "2023-06-20 11:59:07.098499" 12:00:13 } 12:00:13 12:00:13 STDOUT: 12:00:13 12:00:13 Bringing machine 'solaris-11' up with 'virtualbox' provider... 12:00:13 ==> solaris-11: Importing base box 'development/solaris11'... 12:00:13 Progress: 10% Progress: 20% Progress: 30% Progress: 40% Progress: 50% Progress: 60% Progress: 90% 12:00:13 12:00:13 12:00:13 STDERR: 12:00:13 12:00:13 There was an error while executing `VBoxManage`, a CLI used by Vagrant 12:00:13 for controlling VirtualBox. The command and stderr is shown below. 12:00:13 12:00:13 Command: ["import", "/var/root/.vagrant.d/boxes/development-VAGRANTSLASH-solaris11/0/virtualbox/box.ovf", "--vsys", "0", "--vmname", "undefined_solaris_instance_2021-01-19T21:55:40Z_1687255148630_73503", "--vsys", "0", "--unit", "12", "--disk", "/Users/jenkins/VirtualBox VMs/undefined_solaris_instance_2021-01-19T21_55_40Z/box-disk001_1.vmdk"] 12:00:13 12:00:13 Stderr: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100% 12:00:13 Interpreting /var/root/.vagrant.d/boxes/development-VAGRANTSLASH-solaris11/0/virtualbox/box.ovf... 12:00:13 OK. 12:00:13 0%...10%...20%...30%...40%...50%...60%... 12:00:13 Progress state: NS_ERROR_INVALID_ARG 12:00:13 VBoxManage: error: Appliance import failed 12:00:13 VBoxManage: error: Code NS_ERROR_INVALID_ARG (0x80070057) - Invalid argument value (extended info not available) 12:00:13 VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg *)" at line 1119 of file VBoxManageAppliance.cpp 12:00:13 12:00:13 12:00:13 MSG: 12:00:13 12:00:13 non-zero return code 12:00:13 12:00:13 PLAY RECAP ********************************************************************* 12:00:13 QA_agent_solaris11_Test_integration_B40376_20230620104433 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 ```The error with Solaris seemed to be a problem with the node provider, as the last tests launched haven't had the same error.
NFS mounting error
The
agent_test
builds with Ubuntu managers have failed Ubuntu and CentOS agents with an error about mounting the NFS.NFS mounting error
``` fatal: [ID2_B40393_Test_integration_ubuntu_agent]: FAILED! => { 15:22:47 "changed": true, 15:22:47 "cmd": [ 15:22:47 "mount", 15:22:47 "172.31.15.112:/nfs_shared_folder", 15:22:47 "/nfs-mount-point" 15:22:47 ], 15:22:47 "delta": "0:02:05.055252", 15:22:47 "end": "2023-06-20 13:22:47.368805", 15:22:47 "invocation": { 15:22:47 "module_args": { 15:22:47 "_raw_params": "mount 172.31.15.112:/nfs_shared_folder /nfs-mount-point", 15:22:47 "_uses_shell": false, 15:22:47 "argv": null, 15:22:47 "chdir": null, 15:22:47 "creates": null, 15:22:47 "executable": null, 15:22:47 "removes": null, 15:22:47 "stdin": null, 15:22:47 "stdin_add_newline": true, 15:22:47 "strip_empty_ends": true, 15:22:47 "warn": true 15:22:47 } 15:22:47 }, 15:22:47 "rc": 32, 15:22:47 "start": "2023-06-20 13:20:42.313553" 15:22:47 } 15:22:47 15:22:47 STDERR: 15:22:47 15:22:47 mount.nfs: Connection refused 15:22:47 15:22:47 15:22:47 MSG: 15:22:47 15:22:47 non-zero return code ```Error in
Fetch zip HTML report
There is an error which happens apparently randomly on the tests, on task
Fix zip HTML report
, it makes some pipelines fail even though all tests are passed. The pipeline doesn't give any information except that there is an error in the task:Example of the error in one failed build
``` 12:04:00 TASK [Fetch zip HTML report from Windows] ************************************** 12:04:00 task path: /home/ec2-user/workspace/Test_integration@2/quality/tests/integration/test_integration.yaml:135 12:04:00 changed: [ID3_B40444_Test_integration_windows_agent] => { 12:04:00 "changed": true, 12:04:00 "checksum": "0cd60be3d5c24670a8165af1019bcd2cf512fce3", 12:04:00 "dest": "/home/ec2-user/workspace/Test_integration@2/agent_windows_html_report_Test_integration_B40444_20230622102033.zip", 12:04:00 "md5sum": "b96274915bedc4e109644bf759dec942", 12:04:00 "remote_checksum": "0cd60be3d5c24670a8165af1019bcd2cf512fce3", 12:04:00 "remote_md5sum": null 12:04:00 } 12:04:00 12:04:00 PLAY RECAP ********************************************************************* 12:04:00 ID3_B40444_Test_integration_windows_agent : ok=3 changed=3 unreachable=0 failed=1 skipped=8 rescued=0 ignored=0 ```