juliamagan commented 1 week ago

Tests information


Main release stage issue	https://github.com/wazuh/wazuh/issues/23246
Main Vulnerability Detection tests issue	https://github.com/wazuh/wazuh/issues/23298
Version	4.8.0
Release stage	RC 1
Tag	https://github.com/wazuh/wazuh-qa/tree/v4.8.0-rc1
Previous E2E Vulnerability tests issue	https://github.com/wazuh/wazuh/issues/23060

Description

The following issue aims to run all Vulnerability Detector E2E tests for the current release candidate, report the results, and open new issues for any encountered errors.

Instruction

Run the vulnerability detector tests using the Test_e2e_system pipeline:

Conclusion :red_circle:

Report: https://github.com/wazuh/wazuh/files/15234960/Test_e2e_system_284_test_vulnerability_detector.zip

Rebits commented 1 week ago

After talking with @juliamagan we have decided to launch these tests twice, with the v4.8.0-rc1 tag branch and with recent fixes reported in https://github.com/wazuh/wazuh-qa/issues/5319

v4.8.0-rc1: https://ci.wazuh.info/job/Test_e2e_system/280
vd-global-fixes: https://ci.wazuh.info/job/Test_e2e_system/284

E2E tests fail for v4.8.0-rc1 due to https://github.com/wazuh/wazuh-jenkins/issues/6445. This was fixed in vd-global-fixes branch

Rebits commented 1 week ago

Vulnerability Detection E2E Tests :red_circle:

Report: Test_e2e_system_284_test_vulnerability_detector.zip

Failed cases

All test cases fail:

test_first_syscollector_scan[vd_disabled_when_agents_registration
test_first_syscollector_scan[vd_enabled_when_agents_registration]
test_consistency_initial_scans
test_syscollector_second_scan
test_install_vulnerable_package_when_agent_down[install_package]
test_change_agent_manager[install_package]
test_vulnerability_detector_scans_cases (All cases)

General Failures :red_circle:

Broken database. The proposed fix does not solve the issue https://github.com/wazuh/wazuh/pull/23216. The issue was reopened https://github.com/wazuh/wazuh/issues/22847

2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(211): no such table: sys_hwinfo
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(217): no such table: sys_osinfo
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(205): no such table: sys_netiface
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(193): no such table: sys_netproto
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(199): no such table: sys_netaddr
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(175): no such table: sys_programs
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(184): no such table: sys_hotfixes
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(187): no such table: sys_ports
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(87): no such table: sync_info
2024/05/07 09:11:30 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() stmt(169): no such table: sys_processes
2024/05/07 09:11:30 wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot perform range checksum
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_hwinfo
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_osinfo
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_netiface
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:31 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:31 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:32 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:32 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:32 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
2024/05/07 09:12:32 wazuh-db: ERROR: (5214): Null statement on internal cache.
2024/05/07 09:12:32 wazuh-db: ERROR: DB(002) sqlite3_prepare_v2() : no such table: sys_processes
...

Initial Scan Tests :red_circle:

Vulnerability detection does not detect any vulnerability for the centOS agent (agent4) at the initial scans: https://github.com/wazuh/wazuh/issues/23326
Vulnerabilities detected before and after the vulnerability detection module is enabled are not consistent: https://github.com/wazuh/wazuh/issues/23327

Install/Upgrade/Removal cases :red_circle:

No vulnerabilities were triggered in agent agent4 for test_install_vulnerable_package_when_agent_down case. Reported in https://github.com/wazuh/wazuh/issues/23328
No vulnerabilities or alerts were triggered in agents agent3, agent5, agent2 in all cases. Reported in https://github.com/wazuh/wazuh/issues/23328

Evidence collection :red_circle:

The test case test_change_agent_manager[install_package] truncates logs after the agent's manager change, providing very little evidence to troubleshoot failures in this test. Reported in https://github.com/wazuh/wazuh-qa/issues/5345

Environment unstable :red_circle:

In the upgrade_package_nonvulnerable_to_vulnerable test case, it has failed the upgrade of the vulnerable package for Windows agent:

2024-05-07 11:11:32 Error installing the package on agent2: Failed to install the package in agent2: {'changed': False, 'msg': 'Unhandled exception while executing module: Exception calling "OpenPackage" with "2" argument(s): "This installation package could not be opened. Verify that the package exists and that you can access it, or contact the application vendor to verify that this is a valid Windows Installer package"'} (remote_operations_handler.py:401)

Reported in https://github.com/wazuh/wazuh-qa/issues/5344

MARCOSD4 commented 1 week ago

LGTM