search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
65 stars 32 forks source link

Failures in Vulnerability Detection E2E Tests RC 2 #5397

Closed Rebits closed 5 months ago

Rebits commented 6 months ago

Description

The end-to-end (E2E) VD tests are failing. The tests were executed to validate the issue described in wazuh-qa issue #5368, but it seems the tests are not passing.

Initial thoughts attributed the failures to a database error. However, similar issues appeared in wazuh-qa issue #5319, where vulnerabilities were detected. Therefore, there might be an additional underlying problem.

Report details

Observations

macOS agent triggers an OS Vulnerability alert:

{"timestamp":"2024-05-16T12:49:09.430+0000","rule":{"level":7,"description":"CVE-2023-48795 affects macOS"}}

This alert suggests OS updates, which should not occur during the tests.

Analysis

Potential causes for these errors:

Action Plan

Since it is unclear if the database error is solely responsible, and there might be other underlying issues, we propose the following:

Validated by

Conclusion :red_circle:

E2E Tests

Vulnerability Detection

Rebits commented 6 months ago

Testing branch and package creation

Development Branch
qa/tmp-4.8.0-22847-fix

Overview

In order to determine the root of the failures in E2E tests, we have created the temporal branch qa/tmp-4.8.0-22847-fix. This branch comes from 4.8.0 (https://github.com/wazuh/wazuh/commit/678bca8ff3e86158ccdeadc24cb0131c129c28c4) and it includes all the fixes present in 4.7.5 (https://github.com/wazuh/wazuh/commit/2f97b3386e04fdcc1395118159056e447bd6effd)

Packages Creation

For this testing, we have created new packages only for Deb manager and Indexer. No changes were performed in the agents, so no new packages were needed.

[!NOTE] The creation of Indexer packages was necessitated by constraints within the deployment tool utilized for our End-to-End (E2E) tests (https://ci.wazuh.info/job/Wazuh_QA_environment/1166/)

Environment deployment

The testing environment was deployed using the Wazuh_QA_Environment pipeline.

Rebits commented 6 months ago

E2E Vulnerability Detection Tests

[!IMPORTANT] These tests were conducted with the outdated vulnerability index name (#5401). For results after resolving this issue, please check this comment: https://github.com/wazuh/wazuh-qa/issues/5397#issuecomment-2119877559

Report: R1.zip

Reported issues:

Initial scan tests :red_circle:

Initial scan tests fail due to the presence of errors in the managers and the absence of any vulnerabilities in the index:

>       assert test_result.get_test_result(), test_result.report()
E       AssertionError: 
E         Test test_first_syscollector_scan[vd_disabled_when_agents_registration] failed
E         
E         Check all_agents_scanned_syscollector_first_scan succeeded
E         Check all_agents_scanned_vulnerability_first_scan failed. Evidences (['agents_not_scanned_vulnerability_first_scan']) can be found in the report.
E         Check no_errors failed. Evidences (['error_level_messages']) can be found in the report.
E         -----
E         
E       assert False
E        +  where False = <bound method TestResult.get_test_result of <wazuh_testing.end_to_end.TestResult object at 0x7f1d1ea83160>>()
E        +    where <bound method TestResult.get_test_result of <wazuh_testing.end_to_end.TestResult object at 0x7f1d1ea83160>> = <wazuh_testing.end_to_end.TestResult object at 0x7f1d1ea83160>.get_test_result

Reviewing the evidence collected we can see the following errors in the report:

{
    "manager1": {
        "ERROR": [
            "2024/05/17 12:11:11 wazuh-modulesd:vulnerability-scanner[83133] scanOrchestrator.hpp:143 at operator()(): ERROR: Error processing delayed event: Error executing rescan for multiple agents."
        ],
        "CRITICAL": [],
        "WARNING": []
    },
    "manager2": {
        "ERROR": [
            "2024/05/17 12:11:14 wazuh-modulesd:vulnerability-scanner[72528] scanOrchestrator.hpp:143 at operator()(): ERROR: Error processing delayed event: Error executing rescan for multiple agents."
        ],
        "CRITICAL": [],
        "WARNING": []
    },

The Error processing delayed event: Error executing rescan for multiple agents seems to have occurred in both managers.

Vulnerability Index is empty

{
    "agent1": [],
    "agent3": [],
    "agent4": [],
    "agent5": [],
    "agent2": []
}

In addition, we can see that the index isn't even created:

root@ip-172-31-9-51:/home/qa# curl -k -u USER:PASSWORD https://172.31.9.51:9200/wazuh-states-vulnerabilities/_search -H 'Content-Type: application/json' -d '{
  "size": 1000,
  "query": {

    "bool": {
      "must": [
        {
          "match": {
            "agent.id": "001"
          }
        }
      ]
    }
  }
}'
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [wazuh-states-vulnerabilities]","index":"wazuh-states-vulnerabilities","resource.id":"wazuh-states-vulnerabilities","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [wazuh-states-vulnerabilities]","index":"wazuh-states-vulnerabilities","resource.id":"wazuh-states-vulnerabilities","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}root@ip-172-31-9-51:/home/qa#

This was not a configuration error. We can see that initially, indexer-connected was correctly initialized

2024/05/17 12:07:09 indexer-connector[80410] indexerConnector.cpp:319 at initialize(): INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-wazuh.

However, we can see that vulnerabilities were processed:

...
2024/05/17 12:11:10 wazuh-modulesd:vulnerability-scanner[83133] packageScanner.hpp:270 at platformVerify(): DEBUG: The platform is in the list based on CPE comparison for Package: kernel, Version: 4.18.0-348.20.1.el7, CVE: CVE-2023-0160, Content platform CPE: cpe:/o:redhat:enterprise_linux:7 OS CPE: cpe:/o:redhat:enterprise_linux:7
2024/05/17 12:11:10 wazuh-modulesd:vulnerability-scanner[83133] packageScanner.hpp:536 at versionMatch(): DEBUG: No match due to default
...

After this failure, the rest of the tests were aborted to troubleshoot the environment. The vulnerability state was never recovered even after restarting both managers and waiting more than half an hour.

[!IMPORTANT] It seems that tests are failing due to the index has been renamed https://github.com/wazuh/wazuh/pull/23274 Currentl researching impact of detected error in E2E tests

After changing index name initial scans seems to detect the vulnerabilities. However, previously mentioned error was present. Reported in https://github.com/wazuh/wazuh/issues/23512

Rebits commented 6 months ago

Due to the launch tests running in debug mode, obtaining full results takes a significant amount of time. These tests have been automatically triggered using custom packages through the Test_e2e_system pipeline.

We are using a custom branch, tmp-testing-vd-rc2, which incorporates fixes from the following issues: #5401 and #5368. These fixes aim to accurately assess the current status of the VD module.

Rebits commented 6 months ago

E2E Vulnerability Detection Tests (Vulnerability Index rename fix) :red_circle:

This test iteration includes the fix for renaming the vulnerability tests. For more details, see: https://github.com/wazuh/wazuh-qa/issues/5401.

TestInitialScans :red_circle:

Summary

Vulnerability Detection Module

E2E Tests

Details

test_first_syscollector_scan[vd_disabled_when_agents_registration] :red_circle: Vulnerabilities were found for all agents. However, some errors were detected: ``` { "manager1": { "ERROR": [ "2024/05/17 18:27:27 wazuh-modulesd:vulnerability-scanner: ERROR: Error processing delayed event: Error executing rescan for multiple agents." ], "CRITICAL": [], "WARNING": [] }, "manager2": { "ERROR": [], "CRITICAL": [], "WARNING": [ "2024/05/17 18:12:03 wazuh-remoted: WARNING: Agent key already in use: agent ID '005'" ] }, ``` - https://github.com/wazuh/wazuh/issues/23512 - https://github.com/wazuh/wazuh-qa/issues/5403
test_first_syscollector_scan[vd_enabled_when_agents_registration] :red_circle: Vulnerabilities were found for all agents. However, some errors were detected: ``` { "manager1": { "ERROR": [ "2024/05/17 18:27:27 wazuh-modulesd:vulnerability-scanner: ERROR: Error processing delayed event: Error executing rescan for multiple agents." ], "CRITICAL": [], "WARNING": [] }, "manager2": { "ERROR": [], "CRITICAL": [], "WARNING": [ "2024/05/17 18:12:03 wazuh-remoted: WARNING: Agent key already in use: agent ID '005'" ] }, ``` These logs are the same as the ones reported in the `vd_disabled_when_agents_registration` case. In initial tests, no log truncation is performed in order to detect any error that occurred in the test setup phase - https://github.com/wazuh/wazuh/issues/23512 - https://github.com/wazuh/wazuh-qa/issues/5403
test_consistency_initial_scans :red_circle: It appears that the vulnerabilities detected differ between agents registered before the Vulnerability Detection module was enabled and those registered after. Upon reviewing the list of vulnerabilities, we notice some that are present in the first scenario but not in the second. For example: ``` [ "CVE-2023-2426", "vim", "2:8.2.3995-1ubuntu2", "x86_64" ], ``` Upon reviewing the evidence, we found that the test_syscollector_second_scan test includes the missing vulnerabilities. This confirms that the vulnerabilities are detected by the scan, although the specified test duration is insufficient. A pending issue will be opened to address this. - Increase E2E Vulnerability Detection consistency tests timeout. Pending issue to be opened
test_syscollector_second_scan :red_circle: It appears that the vulnerabilities detected differ between syscollector scans. This issue seems to be the same as the one present in the `test_consistency_initial_scans` test. Vulnerabilities not detected in the vd_enabled_when_agents_registration test case are correctly detected in the second scan, likely due to insufficient specified time in the initial test. - https://github.com/wazuh/wazuh-qa/issues/5404

TestScanSyscollectorCases :red_circle:

Summary

Vulnerability Detection Module

E2E Tests

Details

test_install_vulnerable_package_when_agent_down[install_package] :red_circle: Tests fail because some unexpected vulnerabilities appear, and some expected vulnerabilities are not found. ``` E Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report. E Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report. ``` In addition, we can see in the manager 1 alerts unexpected macOS vulnerability alerts affecting the OS: ``` {"timestamp":"2024-05-17T18:51:30.904+0000","rule":{"level":7,"description":"CVE-2023-42914 affects macOS","id":"23504","firedtimes":58,"mail":false,"groups":["vulnerability-detector"],"gdpr":["IV_35.7.d"],"pci_dss":["11.2.1","11.2.3"],"tsc":["CC7.1","CC7.2"]},"agent":{"id":"005","name":"agent6"},"manager":{"name":"ip-172-31-4-9"},"id":"1715971890.3684507","cluster":{"name":"wazuh","node":"master"},"decoder":{"name":"json"},"data":{"vulnerability":{"assigner":"apple","cve":"CVE-2023-42914","cvss":{"cvss3":{"base_score":"6.300000","vector":{"availability":"NONE","confidentiality_impact":"NONE","integrity_impact":"HIGH","privileges_required":"NONE","scope":"CHANGED","user_interaction":"REQUIRED"}}},"enumeration":"CVE","package":{"architecture":"arm64","condition":"Package less than 14.2","name":"macOS","source":"OS","version":"14.0"},"published":"2023-12-12T01:15:12Z","rationale":"The issue was addressed with improved memory handling. This issue is fixed in macOS Sonoma 14.2, iOS 17.2 and iPadOS 17.2, watchOS 10.2, macOS Ventura 13.6.3, tvOS 17.2, iOS 16.7.3 and iPadOS 16.7.3, macOS Monterey 12.7.2. An app may be able to break out of its sandbox.","reference":"http://seclists.org/fulldisclosure/2023/Dec/10, http://seclists.org/fulldisclosure/2023/Dec/11, http://seclists.org/fulldisclosure/2023/Dec/12, http://seclists.org/fulldisclosure/2023/Dec/13, http://seclists.org/fulldisclosure/2023/Dec/7, http://seclists.org/fulldisclosure/2023/Dec/8, http://seclists.org/fulldisclosure/2023/Dec/9, https://support.apple.com/en-us/HT214034, https://support.apple.com/en-us/HT214035, https://support.apple.com/en-us/HT214036, https://support.apple.com/en-us/HT214037, https://support.apple.com/en-us/HT214038, https://support.apple.com/en-us/HT214040, https://support.apple.com/en-us/HT214041","severity":"Medium","status":"Active","title":"CVE-2023-42914 affects macOS","type":"Packages","updated":"2023-12-14T14:54:10Z"}},"location":"vulnerability-detector"} ``` Researching these checks we can see the following issues - `CVE-2023-4822` vulnerability is wrongly expected for Grafana packages. Reported https://github.com/wazuh/intelligence-data/issues/233 - No vulnerabilities were detected for the NodeJS package for the Windows agent. Pending issue to be opened - macOS agent triggers vulnerabilities for initial syscollector scan. Pending issue to be opened
test_change_agent_manager[install_package] :red_circle: Test result not valid due to error introduced in the development https://github.com/wazuh/wazuh-qa/pull/5369. This bug impacted the teardown phase, so packages installed in the previous tests were not uninstalled correctly. This bug affects some of the TestScanSyscollectorCases tests. These tests should be run again with a proper teardown phase. - https://github.com/wazuh/wazuh-qa/issues/5368
test_vulnerability_detector_scans_cases[remove_package] :red_circle: Missing mitigated alerts ``` E Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report. ``` Missing vulnerabilities: ```{ "agent2": [ [ "CVE-2021-4044", "Node.js", "17.0.1", "x86_64" ], [ "CVE-2021-44531", "Node.js", "17.0.1", "x86_64" ], [ "CVE-2021-44532", "Node.js", "17.0.1", "x86_64" ], [ "CVE-2021-44533", "Node.js", "17.0.1", "x86_64" ], [ "CVE-2022-0778", "Node.js", "17.0.1", "x86_64" ], [ "CVE-2022-21824", "Node.js", "17.0.1", "x86_64" ] ] } ``` - No vulnerabilities were detected for the NodeJS package for the Windows agent. Pending issue to be opened
test_vulnerability_detector_scans_cases[upgrade_package_maintain_vulnerability :red_circle: The following checks failed: ``` E Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report. E Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report. E Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report. E Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report. E Check setup_operation_results failed. Evidences (['setup_failed_agents']) can be found in the report. E Check no_duplicated_vulnerabilities failed. Evidences (['duplicated_vulnerabilities']) can be found in the report. ``` - No vulnerabilities were detected for the NodeJS package for the Windows agent. Pending issue to be opened - `CVE-2023-4822` vulnerability is wrongly expected for Grafana packages. Reported https://github.com/wazuh/intelligence-data/issues/233 - `CVE-2023-3128` is not included in vulnerabilities lists for Grafana packages. Fixed in https://github.com/wazuh/wazuh-qa/issues/5368 (b9d6f2b891c67ac89a89bfc07561d7ba70513edc) - Duplicated vulnerabilities found in index. List of vulnerabilities duplicated: ``` [ { "agent1": [ [ "CVE-2022-23552", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-39307", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-36062", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2023-0594", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2023-0507", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2023-2183", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-35957", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-23498", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-31130", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-31097", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2023-3128", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-39324", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-39306", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-39229", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2023-4822", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-39201", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2023-1410", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-31107", "grafana", "8.5.6-1", "x86_64" ], [ "CVE-2022-31123", "grafana", "8.5.6-1", "x86_64" ] ] }, { "agent3": [ [ "CVE-2022-31107", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-35957", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-36062", "grafana", "8.5.6", "x86_64" ], [ "CVE-2023-0594", "grafana", "8.5.6", "x86_64" ], [ "CVE-2023-0507", "grafana", "8.5.6", "x86_64" ], [ "CVE-2023-2183", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-31130", "grafana", "8.5.6", "x86_64" ], [ "CVE-2023-3128", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-31097", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-39324", "grafana", "8.5.6", "x86_64" ], [ "CVE-2023-4822", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-23498", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-39229", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-39306", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-31123", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-23552", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-39201", "grafana", "8.5.6", "x86_64" ], [ "CVE-2023-1410", "grafana", "8.5.6", "x86_64" ], [ "CVE-2022-39307", "grafana", "8.5.6", "x86_64" ] ] }, { "agent4": [ [ "CVE-2022-31107", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-39306", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-31123", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-23552", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-39307", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-36062", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2023-0594", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2023-0507", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2023-2183", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-35957", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-31130", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-31097", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-39324", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2023-3128", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-39229", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2023-4822", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-39201", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2023-1410", "grafana", "8.5.6-1", "arm64" ], [ "CVE-2022-23498", "grafana", "8.5.6-1", "arm64" ] ] }, { "agent5": [ [ "CVE-2022-31107", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-36062", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-35957", "grafana", "8.5.6", "arm64" ], [ "CVE-2023-0594", "grafana", "8.5.6", "arm64" ], [ "CVE-2023-0507", "grafana", "8.5.6", "arm64" ], [ "CVE-2023-2183", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-31130", "grafana", "8.5.6", "arm64" ], [ "CVE-2023-3128", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-31097", "grafana", "8.5.6", "arm64" ], [ "CVE-2023-4822", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-39324", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-23498", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-39306", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-31123", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-39229", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-23552", "grafana", "8.5.6", "arm64" ], [ "CVE-2023-1410", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-39201", "grafana", "8.5.6", "arm64" ], [ "CVE-2022-39307", "grafana", "8.5.6", "arm64" ] ] }, { "agent6": [ [ "CVE-2017-16014", "http-proxy", "0.5.10", "" ] ] } ] ```
test_vulnerability_detector_scans_cases[upgrade_package_add_vulnerability] :red_circle: The following checks failed ``` E Check operation_successfull_for_all_agents succeeded E Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report. E Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report. E Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report. E Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report. ``` - No vulnerabilities were detected for the NodeJS package for the Windows agent. Pending issue to be opened - `CVE-2023-4822` vulnerability is wrongly expected for Grafana packages. Reported https://github.com/wazuh/intelligence-data/issues/233 - Duplicated vulnerabilities found in index. List of vulnerabilities duplicated: ``` [ { "agent1": [ [ "CVE-2022-39201", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2023-1387", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2023-2183", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-39307", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-23498", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-31130", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-39306", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-35957", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-36062", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-39229", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-31123", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2023-4822", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-23552", "grafana", "9.1.1-1", "x86_64" ], [ "CVE-2022-39324", "grafana", "9.1.1-1", "x86_64" ] ] }, { "agent3": [ [ "CVE-2022-23498", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-36062", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-39306", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-31130", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-23552", "grafana", "9.1.1", "x86_64" ], [ "CVE-2023-1387", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-39229", "grafana", "9.1.1", "x86_64" ], [ "CVE-2023-4822", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-31123", "grafana", "9.1.1", "x86_64" ], [ "CVE-2023-2183", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-39307", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-35957", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-39324", "grafana", "9.1.1", "x86_64" ], [ "CVE-2022-39201", "grafana", "9.1.1", "x86_64" ] ] }, { "agent4": [ [ "CVE-2023-2183", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2023-1387", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-39307", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-23498", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-31130", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-39306", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-35957", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-36062", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-39229", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-31123", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2023-4822", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-23552", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-39324", "grafana", "9.1.1-1", "arm64" ], [ "CVE-2022-39201", "grafana", "9.1.1-1", "arm64" ] ] }, { "agent5": [ [ "CVE-2022-23498", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-36062", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-39306", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-31130", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-23552", "grafana", "9.1.1", "arm64" ], [ "CVE-2023-1387", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-39229", "grafana", "9.1.1", "arm64" ], [ "CVE-2023-4822", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-31123", "grafana", "9.1.1", "arm64" ], [ "CVE-2023-2183", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-39307", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-35957", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-39324", "grafana", "9.1.1", "arm64" ], [ "CVE-2022-39201", "grafana", "9.1.1", "arm64" ] ] }, { "agent6": [ [ "CVE-2021-3749", "axios", "0.10.0", "" ], [ "CVE-2020-28168", "axios", "0.10.0", "" ], [ "CVE-2023-45857", "axios", "0.10.0", "" ], [ "CVE-2019-10742", "axios", "0.10.0", "" ] ] } ] ```
test_vulnerability_detector_scans_cases[upgrade_package_maintain_add_vulnerability] :red_circle: The following checks failed: ``` E Check expected_vulnerabilities_found_in_index failed. Evidences (['missing_vulnerabilities']) can be found in the report. E Check no_unexpected_vulnerabilities_found_in_index failed. Evidences (['unexpected_vulnerabilities']) can be found in the report. E Check expected_vulnerability_affected_alert failed. Evidences (['missing_affected_alerts']) can be found in the report. E Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report. ``` - No vulnerabilities were detected for the NodeJS package for the Windows agent. Pending issue to be opened - `CVE-2023-4822` vulnerability is wrongly expected for Grafana packages. Reported https://github.com/wazuh/intelligence-data/issues/233 - Duplicated vulnerabilities found in the index. List of vulnerabilities duplicated: ``` [ { "agent1": [ [ "CVE-2023-0594", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-2183", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2022-39328", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-3128", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2022-23498", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-4822", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2022-39306", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-0507", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2022-23552", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-1387", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-1410", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2023-22462", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2022-39324", "grafana", "9.2.0-1", "x86_64" ], [ "CVE-2022-39307", "grafana", "9.2.0-1", "x86_64" ] ] }, { "agent3": [ [ "CVE-2022-39324", "grafana", "9.2.0", "x86_64" ], [ "CVE-2022-23498", "grafana", "9.2.0", "x86_64" ], [ "CVE-2022-39306", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-0507", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-22462", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-4822", "grafana", "9.2.0", "x86_64" ], [ "CVE-2022-39307", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-0594", "grafana", "9.2.0", "x86_64" ], [ "CVE-2022-39328", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-3128", "grafana", "9.2.0", "x86_64" ], [ "CVE-2022-23552", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-1387", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-1410", "grafana", "9.2.0", "x86_64" ], [ "CVE-2023-2183", "grafana", "9.2.0", "x86_64" ] ] }, { "agent4": [ [ "CVE-2023-22462", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-0594", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-2183", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2022-39307", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2022-39328", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-3128", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2022-23498", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2022-39306", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-0507", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-1387", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-1410", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2023-4822", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2022-23552", "grafana", "9.2.0-1", "arm64" ], [ "CVE-2022-39324", "grafana", "9.2.0-1", "arm64" ] ] }, { "agent5": [ [ "CVE-2022-39324", "grafana", "9.2.0", "arm64" ], [ "CVE-2022-23498", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-22462", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-0507", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-4822", "grafana", "9.2.0", "arm64" ], [ "CVE-2022-39307", "grafana", "9.2.0", "arm64" ], [ "CVE-2022-39306", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-0594", "grafana", "9.2.0", "arm64" ], [ "CVE-2022-39328", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-3128", "grafana", "9.2.0", "arm64" ], [ "CVE-2022-23552", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-1387", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-1410", "grafana", "9.2.0", "arm64" ], [ "CVE-2023-2183", "grafana", "9.2.0", "arm64" ] ] }, { "agent6": [ [ "CVE-2021-21388", "systeminformation", "5.0.0", "" ], [ "CVE-2021-21315", "systeminformation", "5.0.0", "" ], [ "CVE-2023-42810", "systeminformation", "5.0.0", "" ] ] } ] ```
test_vulnerability_detector_scans_cases[upgrade_package_remove_vulnerability] :red_circle: The following checks failed: ``` E Check expected_vulnerability_mitigated_alert failed. Evidences (['missing_mitigated_alerts']) can be found in the report. ``` - No vulnerabilities were detected for the NodeJS package for the Windows agent. Pending issue to be opened
test_vulnerability_detector_scans_cases[upgrade_package_nonvulnerable_to_vulnerable] :red_circle: Test result not valid due to error introduced in the development https://github.com/wazuh/wazuh-qa/pull/5369. This bug impact the teardown phase, so packages installed in the previous tests were not uninstalled correctly. This bug affects some of the TestScanSyscollectorCases tests. These tests should be run again with a proper teardown phase. - https://github.com/wazuh/wazuh-qa/issues/5368
test_vulnerability_detector_scans_cases[upgrade_package_nonvulnerable_to_nonvulnerable] :red_circle: Test fails during the teardown stage: ``` E TypeError: 'NoneType' object is not subscriptable ``` Test result not valid due to error introduced in the development https://github.com/wazuh/wazuh-qa/pull/5369. This bug impact the teardown phase, so packages installed in the previous tests were not uninstalled correctly. This bug affects some of the TestScanSyscollectorCases tests. These tests should be run again with a proper teardown phase. - https://github.com/wazuh/wazuh-qa/issues/5368
test_vulnerability_detector_scans_cases[install_package_non_vulnerable] :red_circle: Test result not valid due to error introduced in the development https://github.com/wazuh/wazuh-qa/pull/5369. This bug impact the teardown phase, so packages installed in the previous tests were not uninstalled correctly. This bug affects some of the TestScanSyscollectorCases tests. These tests should be run again with a proper teardown phase. - https://github.com/wazuh/wazuh-qa/issues/5368
test_vulnerability_detector_scans_cases[remove_non_vulnerable_packge] :red_circle: Test result not valid due to error introduced in the development https://github.com/wazuh/wazuh-qa/pull/5369. This bug impact the teardown phase, so packages installed in the previous tests were not uninstalled correctly. This bug affects some of the TestScanSyscollectorCases tests. These tests should be run again with a proper teardown phase. - https://github.com/wazuh/wazuh-qa/issues/5368
Rebits commented 6 months ago

Currently trying to replicate Duplicated vulnerabilities found in index issue

Rebits commented 5 months ago

Research https://github.com/wazuh/wazuh/issues/23530. This seems to be produced by a test bug. Reported in https://github.com/wazuh/wazuh-qa/issues/5410

Initial scan discrepancies seem to be related to changes in VD content. We should consider avoiding increasing the timeout in https://github.com/wazuh/wazuh-qa/issues/5404. Currently researching the issue

Rebits commented 5 months ago

Some final unexpected failures detected in the last iteration of the tests:

Build: https://ci.wazuh.info/job/Test_e2e_system/289/

Test consistency initial scans

Regarding the analysis performed during https://github.com/wazuh/wazuh/issues/23523, alerts triggered during initial scans were expected due to content updates. This behavior should be taken into account by E2E test (created an issue to perform these changes https://github.com/wazuh/wazuh-qa/issues/5412) However, in the last iteration of the E2E test, it seems that the consistency test between initial scans has failed again without multiple content feed updates:

vd_disabled_when_agents_registration

Manager1

2024/05/21 18:07:27 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process
2024/05/21 18:16:17 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed

Manager2

2024/05/21 18:08:18 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process
2024/05/21 18:20:19 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed

However, the test checks for vulnerabilities much later:

2024-05-21 18:44:35 Checking vulnerabilities in the index (test_vulnerability_detector.py:323)

The agents completed multiple syscollector scans since the feed update finished, so the final vulnerabilities should match the latest feed changes:

2024/05/21 18:21:18 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/21 18:21:20 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/05/21 18:22:21 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/21 18:22:24 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/05/21 18:23:24 wazuh-modulesd:syscollector: INFO: Starting evaluation.

vd_enabled_when_agents_registration No new feed update events occurred, and the agents also performed multiple syscollector scans:

2024/05/21 18:56:29 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/05/21 18:57:30 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/21 18:57:32 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/05/21 18:58:32 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/21 18:58:35 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/05/21 18:59:35 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/05/21 18:59:37 wazuh-modulesd:syscollector: INFO: Evaluation finished.

However, the vulnerabilities have changed between scans. For example, the following vulnerability appeared in the second scan:

[    "CVE-2022-48658",    "kernel",    "4.18.0-348.20.1.el7",    "arm64"]

After discussing with @Dwordcito, it appears this is related to https://github.com/wazuh/wazuh/issues/23482. I'll include this issue in the conclusion and ensure the report is added to the issue thread.

Test change agent manager

After some research, it seems that vulnerabilities are correctly generated for this test, although the specified time is not enough. Reported in https://github.com/wazuh/wazuh-qa/issues/5413

Rebits commented 5 months ago

Some manual testing led to the detection of some issues in the case of cluster rename. Reported in https://github.com/wazuh/wazuh/issues/23540

rafabailon commented 5 months ago

LGTM

juliamagan commented 5 months ago

LGTM