Add Vulnerability Detector System End-to-End syscollector initial scans tests

Deblintrake09 commented 7 months ago

Target version	Related issue	Related PR/dev branch
4.8.0	#4369	https://github.com/wazuh/wazuh-qa/pull/4703

Description

This issue aims to create the basic End-to-end test cases defined in https://github.com/wazuh/wazuh-qa/issues/4531. As all tests defined in the issue are part of the minimal required testing suite, and as tests have tight dependencies, the cases that are part of the basis for all test cases will be done in this issue, so development can be parallelized.

Proposed test cases

[x] E2E-VD-1: First syscollector scan (Rsync)
[x] E2E-VD-2: Subsequent scan (Dbsync) without any installation ...

Considerations

This development is the basis for the rest of the test cases. It should be prioritized and developed ASAP to allow parallelization of development the rest of the tests.
Tests must work on all proposed OSs.

Rebits commented 6 months ago

Found some provision errors:

Windows 11 AMI does not include any python. This could lead into potential errors due to it is set by default a python interpreter that does actually not exist in the host. In addition, after some testing in the environment it seems that it is not possible to use some system testing tool over this Windows 11 hosts.
macOS Agent is not correctly provisioned. More information here

It has been created a basic test structure and some fixture for configure the environment. The current E2E approach has several limitations:

Environment validation does not work for the Vulnerability Detector proposed environment. This should be adapted or disabled by default
Configuration of the environment is implemented appending code block in the ossec.conf. It is difficult to implent complex configurations.

Created class method to configure the environment through YAML files

Rebits commented 6 months ago

Continue creating common fixtures for the basic tests.

Rebits commented 6 months ago

Created first PoC of the E2E-VD-3 tests and common structure for all the basic tests cases

Rebits commented 6 months ago

Development branch
`enhacement/4590-vd-basic-test-cases`

There still some issues regarding the monitoring of the logs. Currently working on it

Rebits commented 6 months ago

Development branch
`enhacement/4590-vd-basic-test-cases`

The Installation of a vulnerable package case was implemented correctly. This was performed following this case structure

- case: "Updating a vulnerable package that remains vulnerable to the same CVE"
  id: "update_vuln_package_remain_vulnerable"
  description: "Updating a vulnerable package that remains vulnerable to the same CVE"
  preconditions:
      tasks:
        - install_package:
            package:
              centos: https://downloads.rclone.org/v1.49.5/rclone-v1.49.5-linux-amd64.rpm
              ubuntu: https://downloads.rclone.org/v1.49.5/rclone-v1.49.5-linux-amd64.deb
      check_alerts:
          centos:
            - event: install_package
              name: "rclone"
              version: "1.49.5"
          ubuntu:
              - event: install_package
                name: "rclone"
                version: "1.49.5"
  body:
    tasks:
      - install_package:
          package:
            centos: https://downloads.rclone.org/v1.50.0/rclone-v1.50.0-linux-amd64.rpm
            ubuntu: https://downloads.rclone.org/v1.50.0/rclone-v1.50.0-linux-amd64.deb
    check_alerts:
        centos:
          - event: upgrade_package
            name: "rclone"
            version: "1.50.0"
        ubuntu:
            - event: upgrade_package
              name: "rclone"
              version: "1.50.0"
  teardown:
    - remove_package:
        package:
          centos: rclone
          ubuntu: rclone

This approach will allow us to include desired cases easily. However this first iteration do not include dependency. The implementation of dynamic dependency could lead into delays in the development

Warning Currently it is only checked that the package installation/removal/upgrade alert was generated in the manager. It would be necessary a development branch to implement a fully functional tests.

Currently working

Support for Windows Endpoint
Implementation in the rest of the tests cases
Creation of a solid vulnerability packages repository

Rebits commented 6 months ago

Refactor tests structure in order to improve readability
Fix Windows endpoint
Include support for ARM endpoints

Rebits commented 6 months ago

Basic Syscollector scan are stable for all the endpoints (AMD architectures)
Refactor monitoring methods and improve readability
Fix framework error for windows package installation

Rebits commented 6 months ago

Fixed minor errors in installation of packages for windows and macOS systems
Fixed minor errors handling different architectures cases
Refactor tests in order to migrate some monitoring logic to fixtures
Working in implementing Vulnerabilities monitoring through API and alerts

Rebits commented 6 months ago

Working in apply dependency for each tests cases
Including indexer alerts logic
Include API method to check state index
Refactor some fixtures

Rebits commented 6 months ago

Meeting with @pereyra-m and @Deblintrake09 in order to clarify some of the preconditions specified in https://github.com/wazuh/wazuh-qa/issues/4531
- Tests should only wait for Syscollector Scans. Alerts should be generated and indexes created intermediately after (Vulnerability Scan interval is deprecated).
- State indices will match https://www.elastic.co/guide/en/ecs/current/ecs-vulnerability.html

Regarding this clarifications it is necessary to:

Remove time wait between Syscollector scan in all agents and vulnerability alerts monitoring for the agent
Include methods to gather information from new state indices

Currently working in tests dependency and fixing minor errors in vulnerability alerts gathering. Changes here: https://github.com/wazuh/wazuh-qa/commit/21176da539f42e496371f8142616e4dcff50ca7c

New ETA 17/11/23 due the following reasons:

Tests dependency did not work as expected. We are currently working on https://github.com/wazuh/wazuh-qa/issues/4590#issuecomment-1799608774 schema to dynamically address test case dependency
ARM hosts package management and monitoring has not been tested yet

Rebits commented 6 months ago

Refactor VD tests in order to improve migration to new framework. Done in https://github.com/wazuh/wazuh-qa/commit/b9f1101992dcb7ecc7b67f9f8a72c322536b1ab3 Also refactor schema in order to check generated vulnerabilities in different ways.

Rebits commented 5 months ago

Created PR with development https://github.com/wazuh/wazuh-qa/pull/4703 It is required further testing

[!NOTE] Current approach implement host handling functionality using different E2E modules. This is a workaround in order to avoid the migration of the QA framework. However it is recommended to follow a class approach to implement those operations.

During testing it has been detected a possible issue handling Vulnerabilities of Windows agents registered in worker nodes. Currently researching this issue. In addition it has been detected that HostMonitor approach for monitoring the alerts generated in the environment is not viable due to the large amount of alerts produced in a multi agent environment. It is suggested to check alerts performing API requests to the indexer server instead.

Rebits commented 5 months ago

Currently testing in real environment It seems that in real testing Vulnerability Scan of agents is waiting longer that the expected: https://ci.wazuh.info/job/Test_e2e_system/186/console It is necessary to analyze result. In addition, it is necessary to include a minor logic to gather information about the environment

Rebits commented 5 months ago

Fixed enrollment configuration error in the agents
Currently working in solving timeout errors:

Testing build: https://ci.wazuh.info/job/Test_e2e_system/189/console

Rebits commented 5 months ago

Working on PR review requested changes

Rebits commented 5 months ago

Moved ETA to 27/11/2023 to apply the rest of the PR comments from @juliamagan

Rebits commented 5 months ago

Apply PR requested changes by @juliamagan

Rebits commented 5 months ago

Moved ETA to 28/11/2023 to allow the revision from @juliamagan

wazuh / wazuh-qa