Add Vulnerability Detector System End-to-End tests - Githubissues

wazuh / wazuh-qa

Wazuh - Quality Assurance

GNU General Public License v2.0

61 stars 30 forks source link

Add Vulnerability Detector System End-to-End tests #4591

Closed Deblintrake09 closed 3 months ago

Deblintrake09 commented 7 months ago

Target version	Related issue	Related PR/dev branch
4.8.0 Alpha 1	#4369

Description

This issue aims to create the End-to-end test cases defined in https://github.com/wazuh/wazuh-qa/issues/4531. This test cases use the basic cases defined in https://github.com/wazuh/wazuh-qa/issues/4590

Proposed test cases

[x] E2E-VD-3: Installation of a vulnerable package
[x] E2E-VD-4: Updating a vulnerable package that remains vulnerable to the same CVE
[x] E2E-VD-5: Updating a vulnerable package that becomes vulnerable to another
[x] E2E-VD-6: Updating a vulnerable package that becomes vulnerable to another CVE and retains the previous one
[x] E2E-VD-7: Updating a vulnerable package that ceases to be vulnerable
[x] E2E-VD-8: Deleting a vulnerable package
[x] E2E-VD-9: Installation of a non-vulnerable package
[x] E2E-VD-10: Updating a non-vulnerable package that becomes vulnerable
[x] E2E-VD-11: Updating a non-vulnerable package that remains non-vulnerable
[x] E2E-VD-12: Deleting a non-vulnerable package
[x] E2E-VD-13: Updating the vulnerability feed with a new applicable vulnerability
[x] E2E-VD-14: Updating the vulnerability feed with a new non-applicable vulnerability
[x] E2E-VD-15: Deleting the states index
[x] E2E-VD-16: Updating the vulnerability feed removes a CVE and causes a package to cease to be vulnerable
[x] Adapt E2E-VD1 and E2E-VD2 to the refactored VD module
[ ] E2E-VD-17: Installation of a vulnerable package when agent is offline
[ ] E2E-VD-18: Enable vulnerability detector when some agents are already registered
[ ] E2E-VD-19: Change agents' manager and install a vulnerable package

Considerations

This tests should use the packages proposed in https://github.com/wazuh/wazuh-qa/issues/4529

Rebits commented 6 months ago

Examining testing scenarios through the provided design at https://github.com/wazuh/wazuh-qa/issues/4590#issuecomment-1799608774. Certain challenges have surfaced concerning the monitoring of alerts across arm architectures. We are actively addressing these issues and working towards resolution.

In addition we are working on including methods for handling indices

Rebits commented 6 months ago

Refactor schema in order to check generated vulnerabilities in different ways. Done in https://github.com/wazuh/wazuh-qa/commit/b9f1101992dcb7ecc7b67f9f8a72c322536b1ab3

Rebits commented 5 months ago

Marked On Hold until https://github.com/wazuh/wazuh-qa/pull/4703 is stable

Rebits commented 5 months ago

Continue with the development using https://github.com/wazuh/wazuh-qa/pull/4703 as base development

Rebits commented 5 months ago

Fixed bug in remove_package for windows endpoint
Refactor test
Implemented VD 3, 4, 5 and 6

It has been detected issues in the Vulnerability detector module regarding the detection of some packages. Further research is required

Rebits commented 5 months ago

Adjusted the ETA to align with @wazuh/data-pirates at https://github.com/wazuh/wazuh/issues/14153 development. Finalizing test modifications to seamlessly integrate with the ongoing development.

Rebits commented 5 months ago

Adapted tests to match changes of VD refactor https://github.com/wazuh/wazuh/issues/14153
- [x] Change configurations in order to include vulnerability-detection and indexer
- [x] Deprecate Vuln Endpoints and include methods to ensure state index is not empty
- [x] Minor changes in callback for waiters in order to adapt to content gathering stage
- [ ] Adapt the rest of tests cases to the common yaml structure
- [x] Adapt final methods for alerts checking in the index state

In addition it is necessary to research some not expected behaviors:

Syscollector seems to missed packages if these are installed in certain time gap in during the scan
Syscollector seems to ignore grafana package for ubuntu system. This should be confirmed for the rest of the systems
Vulnerability Detector seems not work after the refactor. This seems to pull correctly the content, but after that stage this does not work.

Rebits commented 5 months ago

[x] Uploaded packages into a common bucket
[x] Review detect alerts method
[x] Refactor detect alert method in order to take into accound the timestamp
[x] Get feedback from @wazuh/data-pirates regarding the VD functionality in the environment
- @Dwordcito https://github.com/wazuh/wazuh/blob/dev-14153-vulndet-refactor/src/wazuh_modules/vulnerability_scanner/indexer/template/legacy-template.json give us the strucutre of the indices state. We can proceed with the comparision methods regarding this template
[x] Implement indices state validator methods
[ ] Implement the rest of the YAML cases

Rebits commented 5 months ago

[x] Implement the rest of the YAML cases

Marked as blocked until development is over

Rebits commented 5 months ago

Adjusted ETA to December 20, 2023, with approval from @davidjiglesias. This modification is prompted by a development delay now anticipated for December 18: https://github.com/wazuh/wazuh/issues/14153

Rebits commented 5 months ago

Starting the migration of tests to accommodate the recent of the Vulnerability Detector.

Generated packages

https://ci.wazuh.info/job/Packages_builder_tier/3288/console

Meeting with @Dwordcito @davidjiglesias about current status of the VD refactor

Rebits commented 5 months ago

We have successfully conducted the first Proof of Concept (POC) for the testing environment, leveraging the newly refactored Vulnerability Detector. Unfortunately, the analysis has revealed several critical errors that currently hinder the progression of testing.

For further details and a comprehensive overview of the identified issues, please refer to the following GitHub link: https://github.com/wazuh/wazuh/issues/20785.

Rebits commented 4 months ago

Created new packages with latests changes in Vulnerability detector

Build: https://ci.wazuh.info/job/Packages_builder_tier/3347

However, no indices of alert was created

root@ip-172-31-9-35:/home/qa# curl -k -u admin:changeme https://172.31.9.35:9200/_cat/indices?v
health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .opensearch-observability   eCUPM1U6RXiXnioci6-_8g   1   0          0            0       208b           208b
green  open   .plugins-ml-config          RATIwdbhSuCvw0AcMDQs1A   1   0          1            0      3.9kb          3.9kb
green  open   wazuh-alerts-4.x-2023.12.28 gplYSwV1T1SVg35Hy2xicg   3   0       2351            0      3.4mb          3.4mb
green  open   .opendistro_security        MmlFVltMQ22KxT2nJLNg-A   1   0         10            0     64.7kb         64.7kb

In addition no vulnerability alerts were generated. It is necessary to achieve feedback from developer team in order to continue with tests

Rebits commented 4 months ago

New packages were created with https://github.com/wazuh/wazuh/commit/937acfc87d50ca867a75f163b957a461fd5ddef1 changes. However, a segmentation fault has been identified in the current development

https://github.com/wazuh/wazuh/issues/21176

Testing cannot proceed until a stable version is delivered

Rebits commented 4 months ago

https://github.com/wazuh/wazuh/issues/21176 seems to be solved. We can proceed with the development of the tests

Rebits commented 4 months ago

Currently working on tests adaptation

Rebits commented 4 months ago

Improve debug messages
Include methods to handle agent
Include in setup fixture agent removal to avoid known bug https://github.com/wazuh/wazuh/issues/21185
Fixed check_consistency method

Detected bug in decoded alerts regarding the package name. Currently researching
Fixed timeouts and waiters method to adjust the development

Rebits commented 4 months ago

Refactor tests in order to allow continue even if one or multiple agents or stages failed. This change has been motivated due to the high time costing of the deploy/provision, to all the possible information in one test iteration. In addition this change was necessary in order to check the behaviour of all the test stages
Refactor reporting. This was necessary in order to get readable information regarding the environment. Without theses changes developer will have serious difficulties troubleshooting the environment
Improve performance of alert gathering methods.
It should be necessary refactor basic tests due to the complexity that they have been get
ETA for first POC for the first basic tests planned for 15/01/2024

Changes in https://github.com/wazuh/wazuh-qa/commit/52b70beedbeeb978246cbf6fcaca5249424fa8cc

Rebits commented 4 months ago

Working on the development of tools for collecting evidence, aiming to amass comprehensive information that empowers our developer team in effectively debugging the environment.

Rebits commented 4 months ago

Stabilize initial scans tests
Created evidence collection tool
Improve report to allow readability
Fix specific package cases tests tools

Changes can be found in: https://github.com/wazuh/wazuh-qa/commit/f5c96e2a8873497205990417d6cf5ceb7242c497

Currently working on:

Improve readability to specific package tests
Improve logging for specific package tests
Research possible issues with the product in regard to the test result
Research possible consistency error in the product
Improve error resistance to all tests and library to avoid aborting the suite in case of not expected data format
Uncomment the rest of tests

Rebits commented 3 months ago

Recreate packages in order to avoid reported bug that does not allow continue with testing
Refactored install and remove functions
Refactored packages metadata

Rebits commented 3 months ago

Refactored code
Create new update_package function
Replace some packages in order to fulfill with the repository
Detected vulnerable packages not detected by the scan

Pending task

Improve logging messages
Include error resistance
Uncomment rest of cases
Testing in real environment

Rebits commented 3 months ago

Detected that .pkg are not detected by syscollector. This will make all macOS checks fails
Refactored update method
Detected failure in windows package uninstallation. Work in progress
Fix evidence gathering for consistency check tests

Rebits commented 3 months ago

Fixed Windows VLC uninstall process
Fixed macOS installation method
Included all tests cases. Ready to perform a PoC. Planned 31/01/2024

Minor issues to fix:

[x] Include fixture session to check connection to all hosts
[ ] Fix macOS uninstall program
[ ] Fix function to check if any vulnerability were triggered (non vulnerable package installation case)
[x] Update documentation in all methods and modules
[x] Include logic to avoid not handle bad installation
[x] Improve logic in case host OS or architecture provided is not included in the supported tests
[x] Improve logging messages
[x] Refactor libraries to improve performance

davidjiglesias commented 3 months ago

LGTM