search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
65 stars 32 forks source link

System Functionality Testing for Vulnerability Detector Feature #4369

Closed davidjiglesias closed 10 months ago

davidjiglesias commented 1 year ago

Overview

This issue is dedicated to the comprehensive end-to-end functionality system testing of the Vulnerability Detector feature. The aim is to ensure the correct operation of all interconnected components and processes involved in the Vulnerability Detector feature, with a focus on its alerting and state management capabilities. The test coverage spans across multiple operating systems, simulating real-world use and ensuring the robustness of the system across various scenarios.

Feature Architecture and Components

The Vulnerability Detector feature works through continuous system scanning and comparison against a vulnerability feed. It uses Wazuh's syscollector to gather software inventory, which is then cross-referenced with the feed.

The architecture includes:

  1. Syscollector module: This is the input source that gathers the software inventory data from the Wazuh agent.
  2. Vulnerability detector module: This module is the core of the feature, which takes the information from Syscollector and cross-references it with the vulnerability feed.
  3. Vulnerability feed: This acts as a reference for known vulnerabilities. Any updates or modifications to this feed can trigger vulnerability detection.
  4. State index: This component tracks the system's vulnerability statuses and alerts, with state changes being driven by events such as software installation, deletion, system scans, or updates to the vulnerability feed.
  5. Alerts index: This manages alerts generated based on changes detected in the system state by the Wazuh Engine through the Vulnerability detector module. These alerts track status changes.

Test Design

The test design ensures that all components work as intended in an integrated, real-world context. We aim to ensure that the Vulnerability Detector feature behaves reliably, issuing appropriate alerts and maintaining accurate state information across various scenarios.

TBD

Chosen Families

Initial Coverage OS

This list will be updated accordingly following the new compatibility matrix and tiers system.

Test Cases

Trigger/Condition Preconditions Expected Outcome Type
First syscollector scan (Rsync) TBD System states and initial vulnerability alerts recorded ("added" only) Event driven
Subsequent scan (Dbsync) without any installation TBD System states and alerts remain unchanged Time driven
Installation of a vulnerable package TBD New entry appears in system states and a vulnerability "added" alert triggered Time driven
Installation of a non-vulnerable package TBD System states and alerts remain unchanged Time driven
Updating the vulnerability feed with a new applicable vulnerability TBD New entry appears in system states and a vulnerability "added" alert is triggered Event driven
Updating the vulnerability feed with a new non-applicable vulnerability TBD System states and alerts remain unchanged Event driven
Deleting the states index TBD The state index needs to be regenerated Event driven
Deleting a non-vulnerable package TBD System states and alerts remain unchanged Time driven
Deleting a vulnerable package TBD Entry of the vulnerable package disappears from the state and a "fixed" alert is triggered Time driven
Updating the vulnerability feed removes a CVE and causes agent to cease to be vulnerable TBD The applicable CVE disappears from the state and a "fixed" alert is triggered Event driven
Updating a vulnerable package that remains vulnerable to the same CVE TBD The version changes in the states index and two alerts are generated (one "fixed" and another "added") Time driven
Updating a vulnerable package that becomes vulnerable to another CVE TBD The version and the CVE change in the states index and two alerts are generated (one "fixed" and another "added") Time driven
Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one TBD The version changes for the existing CVE and a new CVE appears in the states index, with three alerts generated (one "fixed" and two "added") Time driven
Updating a vulnerable package that ceases to be vulnerable TBD The entry for the package disappears from the states index and a "fixed" alert is triggered Time driven
Updating a non-vulnerable package that becomes vulnerable TBD A new state appears in the states index and an "added" alert is triggered Time driven
Updating a non-vulnerable package that remains non-vulnerable TBD System states and alerts remain unchanged Time driven

Test Execution

Security Implications:

Performance Expectations:

Edge Cases/Exception Cases:

Regression Scenarios:

Tasks

Testing environment

Framework

Testing requirements

Tests Development

CI

Rebits commented 1 year ago

We need to expand the requirements because it's currently unclear which architecture the tests should be conducted on. For more details, please refer to the following link: https://github.com/wazuh/wazuh-jenkins/issues/5774.

Furthermore, the precise infrastructure in which the tests should be carried out remains unclear. We need additional information, such as whether it should be on a single node, a multi-node manager setup, an indexer multinode configuration, or another specific setup. To proceed, we require further details. More information and a infrastructure suggestion can be found in https://github.com/wazuh/wazuh-qa/issues/4582

Finally, is necessary to specify the list of the CI process where this tests will be included. This is necessary to estimate the https://github.com/wazuh/wazuh-qa/issues/4589 task

Rebits commented 1 year ago

Meeting with @davidjiglesias confirming the following points related to the infrastrucutre:

Rebits commented 1 year ago

In today's meeting, attended by @Deblintrake09, @juliamagan, @davidjiglesias, and @Rebits, a resolution was reached under the guidance of @davidjiglesias. It has been decided to forego support for macOS Sonoma in our testing environments, citing complications in its integration. Instead, our focus will shift to supporting macOS Ventura ARM.

The details of this decision and the transition plan will be documented in the corresponding issue for further reference and implementation: https://github.com/wazuh/wazuh-qa/issues/4737

Rebits commented 12 months ago

Adjusted ETA to December 20, 2023, with approval from @davidjiglesias.

This modification is prompted by a development delay now anticipated for December 18: https://github.com/wazuh/wazuh/issues/14153

Rebits commented 11 months ago

Testing cannot proceed at the moment due to functionality issues identified in https://github.com/wazuh/wazuh/issues/20785. We will resume development once these issues are addressed.

Rebits commented 11 months ago

A segmentation fault has been identified in the current environment status

Unfortunately, testing cannot proceed until a stable version is delivered.

We have decided to exclude https://github.com/wazuh/internal-devel-requests/issues/472 from the development. The current approach involves utilizing public IP addresses instead, as it helps prevent errors in VPN handling. More information in https://github.com/wazuh/wazuh-jenkins/issues/6041

Rebits commented 10 months ago

https://github.com/wazuh/wazuh/issues/21176 seems to be solved. We can proceed with the development of the tests

davidjiglesias commented 10 months ago

LGTM

Rebits commented 9 months ago

Missing tests cases included in the middle of the development will be tracked in https://github.com/wazuh/wazuh-qa/issues/4914