Closed QU3B1M closed 1 month ago
Created a small library to handle WazuhAPI requests and responses with the purpose of simplifying the API testing for current and future tests. The library contains three files and for now is included inside the tests helpers, later, if its approved it could be configured to be an installable library.
Files:
endpoints.py
- Lists all the WazuhAPI endpoints used by the library
exceptions.py
- Defines exceptions to be raised by the library corresponding to failing status codes
api.py
- The main file of the library where the class WazuhAPI resides.
check-files
on test Install:In the current approach the provision module is in charge of installing the wazuh-agent
, so there is no easy way to take a snapshot of the system files before and after the installation to validate for changes.
If we want to maintain the test "simple" and "modularized" to only do validations, we won't be able to do the check-files, at least with the desired approach.
All test LGTM
The item Test install - Add checkfiles close-word
will be analyzed and discussed with @davidjiglesias and then the appropriate implementation since the current implementation is not valid.
In the evaluation and development of the best model for performing checkfiles
First of all, it was determined in the discussion with the team that the installation and uninstallation tests should consider the installation and uninstallation of components within the test module itself.
On the other hand, we will investigate how to instrument the checkfile.
It is clear that to achieve the check-file, we will have to take a snapshot of the directories and files before and after the 'action'.
Reference: [] : fixed positions. The rest of the test can change the order.
Install
and uninstall
should be the start and the end of the sequence.
If the workflow does not require the install
and the register
, this should be done implicitly by the provision module
without the validations of the stages.
In case the install
and the register
are not required by the user, the test will fail due to a lack of agent installation and registration.
Agent:
Note: The main issue with this is that the installation of pytest will have to be handled by external code outside the test file.
This issue was fixed by adding setup steps to playbooks/setup.yml
Working with a file and directory counter in a large directory can pose challenges. Other modifications may result in alterations to the count, leading to the possibility of false negatives in the test.
It will be necessary to establish pre and post-installation validation criteria.
After communicating with @fcaffieri , it was defined that the Filecheck should be global.
Doing some research, the following method was found:
Still some issues around file reading happened, however, the method is detecting the changes in a specific directory
Some changes have been done in the check-file comparison function:
Install scenario:
postinstall = {'changed_files': ['/var/log/dpkg.log', '/var/log/auth.log', '/var/log/syslog', '/var/log/journal/ea8d563452674de897687e4c554abfba/system.journal', '/var/lib/dpkg/status', '/var/cache/apt/pkgcache.bin'], 'added_files': ['/var/lib/dpkg/info/wazuh-agent.postinst', '/var/lib/dpkg/info/wazuh-agent.conffiles', '/var/lib/dpkg/info/wazuh-agent.preinst', '/var/lib/dpkg/info/wazuh-agent.postrm', '/var/lib/dpkg/info/wazuh-agent.templates', '/var/lib/dpkg/info/wazuh-agent.prerm', '/var/lib/dpkg/info/wazuh-agent.shlibs', '/var/lib/dpkg/info/wazuh-agent.list', '/var/lib/dpkg/info/wazuh-agent.md5sums', '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml', '/var/ossec/wodles/utils.py', '/var/ossec/wodles/__init__.py', '/var/ossec/wodles/aws/aws-s3', '/var/ossec/wodles/azure/orm.py', '/var/ossec/wodles/azure/azure-logs', '/var/ossec/wodles/docker/DockerListener', '/var/ossec/wodles/gcloud/exceptions.py', '/var/ossec/wodles/gcloud/integration.py', '/var/ossec/wodles/gcloud/tools.py', '/var/ossec/wodles/gcloud/gcloud', '/var/ossec/wodles/gcloud/pubsub/subscriber.py', '/var/ossec/wodles/gcloud/buckets/bucket.py', '/var/ossec/wodles/gcloud/buckets/access_logs.py', '/var/ossec/var/run/wazuh-syscheckd-120670.pid', '/var/ossec/var/run/wazuh-agentd.state', '/var/ossec/var/run/wazuh-agentd-120657.pid', '/var/ossec/var/run/wazuh-execd-120646.pid', '/var/ossec/var/run/wazuh-modulesd-120700.pid', '/var/ossec/var/run/wazuh-logcollector-120683.pid', '/var/ossec/var/selinux/wazuh.pp', '/var/ossec/agentless/register_host.sh', '/var/ossec/agentless/ssh_integrity_check_linux', '/var/ossec/agentless/ssh_asa-fwsmconfig_diff', '/var/ossec/agentless/sshlogin.exp', '/var/ossec/agentless/ssh_foundry_diff', '/var/ossec/agentless/ssh_generic_diff', '/var/ossec/agentless/main.exp', '/var/ossec/agentless/su.exp', '/var/ossec/agentless/ssh.exp', '/var/ossec/agentless/ssh_pixconfig_diff', '/var/ossec/agentless/ssh_nopass.exp', '/var/ossec/agentless/ssh_integrity_check_bsd', '/var/ossec/bin/wazuh-control', '/var/ossec/bin/agent-auth', '/var/ossec/bin/wazuh-agentd', '/var/ossec/bin/wazuh-logcollector', '/var/ossec/bin/wazuh-modulesd', '/var/ossec/bin/manage_agents', '/var/ossec/bin/wazuh-syscheckd', '/var/ossec/bin/wazuh-execd', '/var/ossec/lib/libsysinfo.so', '/var/ossec/lib/libstdc++.so.6', '/var/ossec/lib/libwazuhshared.so', '/var/ossec/lib/libfimdb.so', '/var/ossec/lib/libsyscollector.so', '/var/ossec/lib/librsync.so', '/var/ossec/lib/libwazuhext.so', '/var/ossec/lib/libgcc_s.so.1', '/var/ossec/lib/libdbsync.so', '/var/ossec/etc/client.keys', '/var/ossec/etc/localtime', '/var/ossec/etc/wpk_root.pem', '/var/ossec/etc/local_internal_options.conf', '/var/ossec/etc/internal_options.conf', '/var/ossec/etc/ossec.conf', '/var/ossec/etc/shared/rootkit_trojans.txt', '/var/ossec/etc/shared/cis_win2012r2_memberL2_rcl.txt', '/var/ossec/etc/shared/win_audit_rcl.txt', '/var/ossec/etc/shared/cis_rhel_linux_rcl.txt', '/var/ossec/etc/shared/cis_mysql5-6_community_rcl.txt', '/var/ossec/etc/shared/win_malware_rcl.txt', '/var/ossec/etc/shared/cis_sles11_linux_rcl.txt', '/var/ossec/etc/shared/cis_win2012r2_domainL2_rcl.txt', '/var/ossec/etc/shared/cis_mysql5-6_enterprise_rcl.txt', '/var/ossec/etc/shared/cis_rhel5_linux_rcl.txt', '/var/ossec/etc/shared/system_audit_ssh.txt', '/var/ossec/etc/shared/cis_sles12_linux_rcl.txt', '/var/ossec/etc/shared/cis_apache2224_rcl.txt', '/var/ossec/etc/shared/cis_rhel7_linux_rcl.txt', '/var/ossec/etc/shared/win_applications_rcl.txt', '/var/ossec/etc/shared/system_audit_rcl.txt', '/var/ossec/etc/shared/cis_win2012r2_domainL1_rcl.txt', '/var/ossec/etc/shared/cis_debian_linux_rcl.txt', '/var/ossec/etc/shared/cis_rhel6_linux_rcl.txt', '/var/ossec/etc/shared/cis_win2012r2_memberL1_rcl.txt', '/var/ossec/etc/shared/rootkit_files.txt', '/var/ossec/active-response/bin/firewall-drop', '/var/ossec/active-response/bin/kaspersky.py', '/var/ossec/active-response/bin/route-null', '/var/ossec/active-response/bin/restart-wazuh', '/var/ossec/active-response/bin/ip-customblock', '/var/ossec/active-response/bin/firewalld-drop', '/var/ossec/active-response/bin/default-firewall-drop', '/var/ossec/active-response/bin/pf', '/var/ossec/active-response/bin/restart.sh', '/var/ossec/active-response/bin/host-deny', '/var/ossec/active-response/bin/ipfw', '/var/ossec/active-response/bin/disable-account', '/var/ossec/active-response/bin/wazuh-slack', '/var/ossec/active-response/bin/npf', '/var/ossec/active-response/bin/kaspersky', '/var/ossec/logs/active-responses.log', '/var/ossec/logs/ossec.log', '/var/ossec/queue/syscollector/norm_config.json', '/var/ossec/queue/syscollector/db/local.db-journal', '/var/ossec/queue/syscollector/db/local.db', '/var/ossec/queue/fim/db/fim.db', '/var/ossec/queue/fim/db/fim.db-journal', '/var/ossec/queue/sockets/wmodules', '/var/ossec/queue/sockets/control', '/var/ossec/queue/sockets/com', '/var/ossec/queue/sockets/.wait', '/var/ossec/queue/sockets/upgrade', '/var/ossec/queue/sockets/queue', '/var/ossec/queue/sockets/logcollector', '/var/ossec/queue/sockets/syscheck', '/var/ossec/queue/alerts/cfgaq', '/var/ossec/queue/alerts/execq', '/var/ossec/queue/logcollector/file_status.json'], 'deleted_files': []}
Number of files added:
print(len(postinstall['added_files']))
Print results
119
Uninstall scenario:
postdelete = {'changed_files': ['/var/log/dpkg.log', '/var/log/auth.log', '/var/log/syslog', '/var/log/apt/history.log', '/var/log/apt/term.log', '/var/log/journal/ea8d563452674de897687e4c554abfba/system.journal', '/var/lib/dpkg/status', '/var/cache/apt/pkgcache.bin'], 'added_files': [], 'deleted_files': ['/var/lib/dpkg/info/wazuh-agent.postinst', '/var/lib/dpkg/info/wazuh-agent.conffiles', '/var/lib/dpkg/info/wazuh-agent.preinst', '/var/lib/dpkg/info/wazuh-agent.postrm', '/var/lib/dpkg/info/wazuh-agent.templates', '/var/lib/dpkg/info/wazuh-agent.prerm', '/var/lib/dpkg/info/wazuh-agent.shlibs', '/var/lib/dpkg/info/wazuh-agent.list', '/var/lib/dpkg/info/wazuh-agent.md5sums', '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml', '/var/ossec/wodles/utils.py', '/var/ossec/wodles/__init__.py', '/var/ossec/wodles/aws/aws-s3', '/var/ossec/wodles/azure/orm.py', '/var/ossec/wodles/azure/azure-logs', '/var/ossec/wodles/docker/DockerListener', '/var/ossec/wodles/gcloud/exceptions.py', '/var/ossec/wodles/gcloud/integration.py', '/var/ossec/wodles/gcloud/tools.py', '/var/ossec/wodles/gcloud/gcloud', '/var/ossec/wodles/gcloud/pubsub/subscriber.py', '/var/ossec/wodles/gcloud/buckets/bucket.py', '/var/ossec/wodles/gcloud/buckets/access_logs.py', '/var/ossec/var/run/wazuh-syscheckd-120670.pid', '/var/ossec/var/run/wazuh-agentd.state', '/var/ossec/var/run/wazuh-agentd-120657.pid', '/var/ossec/var/run/wazuh-execd-120646.pid', '/var/ossec/var/run/wazuh-modulesd-120700.pid', '/var/ossec/var/run/wazuh-logcollector-120683.pid', '/var/ossec/var/run/wazuh-logcollector.state', '/var/ossec/var/selinux/wazuh.pp', '/var/ossec/agentless/register_host.sh', '/var/ossec/agentless/ssh_integrity_check_linux', '/var/ossec/agentless/ssh_asa-fwsmconfig_diff', '/var/ossec/agentless/sshlogin.exp', '/var/ossec/agentless/ssh_foundry_diff', '/var/ossec/agentless/ssh_generic_diff', '/var/ossec/agentless/main.exp', '/var/ossec/agentless/su.exp', '/var/ossec/agentless/ssh.exp', '/var/ossec/agentless/ssh_pixconfig_diff', '/var/ossec/agentless/ssh_nopass.exp', '/var/ossec/agentless/ssh_integrity_check_bsd', '/var/ossec/bin/wazuh-control', '/var/ossec/bin/agent-auth', '/var/ossec/bin/wazuh-agentd', '/var/ossec/bin/wazuh-logcollector', '/var/ossec/bin/wazuh-modulesd', '/var/ossec/bin/manage_agents', '/var/ossec/bin/wazuh-syscheckd', '/var/ossec/bin/wazuh-execd', '/var/ossec/lib/libsysinfo.so', '/var/ossec/lib/libstdc++.so.6', '/var/ossec/lib/libwazuhshared.so', '/var/ossec/lib/libfimdb.so', '/var/ossec/lib/libsyscollector.so', '/var/ossec/lib/librsync.so', '/var/ossec/lib/libwazuhext.so', '/var/ossec/lib/libgcc_s.so.1', '/var/ossec/lib/libdbsync.so', '/var/ossec/etc/client.keys', '/var/ossec/etc/localtime', '/var/ossec/etc/wpk_root.pem', '/var/ossec/etc/local_internal_options.conf', '/var/ossec/etc/internal_options.conf', '/var/ossec/etc/ossec.conf', '/var/ossec/etc/shared/rootkit_trojans.txt', '/var/ossec/etc/shared/cis_win2012r2_memberL2_rcl.txt', '/var/ossec/etc/shared/win_audit_rcl.txt', '/var/ossec/etc/shared/cis_rhel_linux_rcl.txt', '/var/ossec/etc/shared/cis_mysql5-6_community_rcl.txt', '/var/ossec/etc/shared/win_malware_rcl.txt', '/var/ossec/etc/shared/cis_sles11_linux_rcl.txt', '/var/ossec/etc/shared/cis_win2012r2_domainL2_rcl.txt', '/var/ossec/etc/shared/cis_mysql5-6_enterprise_rcl.txt', '/var/ossec/etc/shared/cis_rhel5_linux_rcl.txt', '/var/ossec/etc/shared/system_audit_ssh.txt', '/var/ossec/etc/shared/cis_sles12_linux_rcl.txt', '/var/ossec/etc/shared/cis_apache2224_rcl.txt', '/var/ossec/etc/shared/cis_rhel7_linux_rcl.txt', '/var/ossec/etc/shared/win_applications_rcl.txt', '/var/ossec/etc/shared/system_audit_rcl.txt', '/var/ossec/etc/shared/cis_win2012r2_domainL1_rcl.txt', '/var/ossec/etc/shared/cis_debian_linux_rcl.txt', '/var/ossec/etc/shared/cis_rhel6_linux_rcl.txt', '/var/ossec/etc/shared/cis_win2012r2_memberL1_rcl.txt', '/var/ossec/etc/shared/rootkit_files.txt', '/var/ossec/active-response/bin/firewall-drop', '/var/ossec/active-response/bin/kaspersky.py', '/var/ossec/active-response/bin/route-null', '/var/ossec/active-response/bin/restart-wazuh', '/var/ossec/active-response/bin/ip-customblock', '/var/ossec/active-response/bin/firewalld-drop', '/var/ossec/active-response/bin/default-firewall-drop', '/var/ossec/active-response/bin/pf', '/var/ossec/active-response/bin/restart.sh', '/var/ossec/active-response/bin/host-deny', '/var/ossec/active-response/bin/ipfw', '/var/ossec/active-response/bin/disable-account', '/var/ossec/active-response/bin/wazuh-slack', '/var/ossec/active-response/bin/npf', '/var/ossec/active-response/bin/kaspersky', '/var/ossec/logs/active-responses.log', '/var/ossec/logs/ossec.log', '/var/ossec/queue/syscollector/norm_config.json', '/var/ossec/queue/syscollector/db/local.db-journal', '/var/ossec/queue/syscollector/db/local.db', '/var/ossec/queue/fim/db/fim.db', '/var/ossec/queue/fim/db/fim.db-journal', '/var/ossec/queue/sockets/wmodules', '/var/ossec/queue/sockets/control', '/var/ossec/queue/sockets/com', '/var/ossec/queue/sockets/.wait', '/var/ossec/queue/sockets/upgrade', '/var/ossec/queue/sockets/queue', '/var/ossec/queue/sockets/logcollector', '/var/ossec/queue/sockets/syscheck', '/var/ossec/queue/alerts/cfgaq', '/var/ossec/queue/alerts/execq', '/var/ossec/queue/logcollector/file_status.json']}
Number of files deleted:
print(len(postdelete['deleted_files'])
Print results
120
This would make us understand that using the number of files is not a good idea. The criteria to be used must be defined.
Moving to an issue with higher priority.
Considering the highlighted point in https://github.com/wazuh/wazuh-qa/issues/4843#issuecomment-1941938284, depending on the test, the provision should receive variables from installing only the manager, only the dependencies and libraries or those tasks should be handled by the test module as well. This point should be discussed with the team.
Another problem is that the test module commands are executed in the agent. In case I want to install the manager from the test module, I will need to refactor the structure of the module
After some discussion with the team. The manager will be installed by the provision. The agent by the Test module.
In case the test module does not include the install
test, the provision module should be added in the Yaml fixture to have the agent provision.
Created install and uninstall + file verification system.
Regarding the check-file, some changes were applied:
It only remains to add the new parameters required for the installation and uninstallation methods
The test_uninstall should always go in the last position of the tests executed on the fixture
- tests: "...,uninstall"
The tests work correctly Correcting and improving script quality Running tests from workflow_engine
After having a meeting with @fcaffieri and @QU3B1M, we were able to conclude that the dependencies should be provisioned by the agent's provisioner, so the agent provisioning should always be executed. It will be worked on at https://github.com/wazuh/wazuh-qa/issues/4859 In the meantime, there will be a patch in deployability/playbooks/tests/setup.yml where it will be provisioned so that the tests developed in this issue can be tested.
Results:
Changes done
The merge between the old and new branches remained pending
Testing Merge
Many tests are failing :red_circle:
After discussing with @QU3B1M, we've gained insights into some issues related to API functions. Consequently, certain changes are necessary regarding the parameters passed to the test. (Many of them are failing due to the change from dependency_ip to dependencies)
On another note:
manager_ip
will be automatically added. If the parameter is set to false, the installation will proceed with MANAGER_IP
instead of the actual IP.current_ip
. If the IP is set to MANAGER_IP
, it will be replaced with the current IP. If a real IP is provided, the test will only trigger a restart.connection was merged to registration
install :green_circle: stop :green_circle: uninstall :green_circle: basic_info :red_circle: restart :red_circle: registration :construction:
Failures
utils.get_client_keys() and api requests
Installation of the agent is done, however the manager is not detecting the installation Even running it manually
For that reason, clientkeys are empy and api is failing getting the agent_id
Naming of the agent was the problem
Tests are all working, however, testing: Installation + Register => some tests in register are failing.
This failures are related to the time that the agent is changing its status and clientkey.
Some dynamic waits should be added.
Dynamic wait added, all tests are running ok
Some points to keep in mind:
If the list of tests does not include install, the agent must be provisioned from provisioning
Otherwise, if it is installed, it will need to go first.
Register registers the agent, if one_line = False, the register will change the MANAGER_IP to the corresponding ip and restart. In case of one_line = True, only the system will be rebooted
Uninstall should go last in case it is executed since it uninstalls the system.
Stop does stop, but a restart has been added at the end of the test
If a test series was run and the manager contains the agent information. If you execute register on that manager with the agent registration, it may fail, since the agent must be deleted from the manager. Otherwise the clientkey may take time (by product design) to replace the dead agent with the new one with the same name and the same ip
Once https://github.com/wazuh/wazuh-qa/issues/4859 is completed, where the provisioning of the agent includes the installation of the dependencies, the modules/testing/setup.yml file must be restored
Requested fixes done and tested
LGTM!
Some error handling test will be done
Testing Error logs:
N | Error | Log | Status |
---|---|---|---|
1 | Wrong inventory | pydantic_core._pydantic_core.ValidationError: 1 validation error for InputPayload | :green_circle: |
2 | Wrong component | main.py: error: argument --component: invalid choice: 'agento' (choose from 'manager', 'agent') | :green_circle: |
3 | Wrong dependencies entry | FileNotFoundError: File "/tmp/dtt1-poc/no_manager-linux-ubuntu-22.04-amd64/inventory.yaml" not found. | :green_circle: |
4 | Wrong live | pydantic_core._pydantic_core.ValidationError: 1 validation error for InputPayload live | :green_circle: |
5 | Wrong one_line | pydantic_core._pydantic_core.ValidationError: 1 validation error for InputPayload one_line | :green_circle: |
6 | Wrong test Wazuh-version/revision | Error is present, but not correctly handled | :red_circle: |
7 | Wrong test name | stderr: 'ERROR: file or directory not found: test_agent/test_instal.py' | :green_circle: |
Adding Error handling and logs to installer, uninstaller and host information
Fixes | N | Error | Log | Status |
---|---|---|---|---|
6 | Wrong Wazuh-version/revision | ERROR Testing:installer.py:55 Error running 'wget' or 'install' command: Command 'wget https://packages-dev.wazuh.com/pre-release/apt/pool/main/w/wazuh-agent/wazuh-agent_4.7.2-1123213_amd64.deb | :green_circle: |
Logs were added in the Testing engine and they are handling the errors correctly
Install
After obtaining all the files generated in var and comparing them with the pre install files, we could see that (except for those that contained ossec or wazuh in their path)
Same additional filters will be added to https://github.com/wazuh/wazuh-qa/issues/4844#issuecomment-1983070759
New changes are required due to the new requirements.
Tests were created
During testing, some additional filters for checkfiles were required in uninstall process.
Those filters were in /boot directory
filter_data = {'/boot': {'added': ['grub2', 'loader', 'vmlinuz', 'System.map', 'config-', 'initramfs'], 'removed': [], 'modified': ['grubenv']}, '/usr/bin': {'added': ['filebeat'], 'removed': [], 'modified': []}, '/root': {'added': ['trustdb.gpg'], 'removed': [], 'modified': []}, '/usr/sbin': {'added': [], 'removed': [], 'modified': []}}
Tests results Vagrant
Os | Result | Adjustments |
---|---|---|
ubuntu 16.04 | :green_circle: | - |
ubuntu 18.04 | :green_circle: | - |
ubuntu 20.04 | :green_circle: | - |
ubuntu 22.04 | :green_circle: | - |
debian 9 | :green_circle: | - |
debian 10 | :green_circle: | - |
debian 11 | :green_circle: | - |
debian 12 | :green_circle: | - |
centos 7 | :green_circle: | - |
centos 8 | :green_circle: | - |
redhat 7 | :green_circle: | - |
redhat 8 | :green_circle: | - |
redhat 9 | :green_circle: | - |
oracle 9 | :green_circle: | Filters added |
amazon2 | :green_circle: | - |
amazon2023 | :white_circle: | No vagrant image |
opensuse 15 | :green_circle: | zyp uninstallation added |
suse 15 | :white_circle: | No vagrant image |
Multiple agents and 1 master connected :green_circle:
Testing in AWS
Some issues doing ssh to EC2 instances
This point will be discussed with @fcaffieri @QU3B1M and @wazuh/devel-devops team
Meet defined: Sync allocation module
- Wednesday, March 20⋅8:00 – 8:30pm
ESP
Changed status to Blocked for third-party
Testing in AWS
raise
instead of sys.exit
issue
parameter does not come in the call to the module, the instance_name
or some other parameter will be used to create the name of the key.Tests results in AWS
OS | Test result | Additional data |
---|---|---|
redhat9 | :green_circle: | |
redhat7 | :red_circle: | AMI failure |
redhat8 | :green_circle: | |
centos7 | :green_circle: | |
centos8 | :green_circle: | |
debian10 | :green_circle: | |
debian11 | :green_circle: | |
debian12 | :green_circle: | |
ubuntu20.04 | :green_circle: | |
ubuntu22.04 | :green_circle: | |
oracle9 | :green_circle: | |
amazon2 | :green_circle: | |
amazon2023 | :green_circle: | |
opensuse | :red_circle: | AMI failure |
suse15 | :green_circle: |
Changes done. Moved to pending review
LGTM
Description
This issue aims to improve and complete the wazuh-agent tests generated in the first iterations of DTT1
Tasks