Deployability testing tier2

[davidjiglesias]

Description

The goal of this issue is to perform an exhaustive testing of Wazuh packages upgrade processes on tier 1 operating systems and architectures. This iteration will build upon the deployability testing established in the first tier (DTT1), now referred to as Deployability Testing Tier 2 (DTT2). DTT2 will concentrate on the upgrade process, ensuring that it is seamless, maintains configurations and permissions, and does not introduce regressions.

Functional requirements

DTT2 focuses on upgrading the following combination of operating systems, versions, and architectures:

| Operating System | Version | Component | Architectures | |----------------------------|------------------------|------------------------------------|-------------------------| | RedHat | 7 | agents, central components | x86_64, aarch64 | | RedHat | 8 | agents, central components | x86_64, aarch64 | | RedHat | 9 | agents, central components | x86_64, aarch64 | | CentOS | 7 | agents, central components | x86_64, aarch64 | | CentOS | 8 | agents, central components | x86_64, aarch64 | | Debian | 10 | agents, central components | x86_64, aarch64 | | Debian | 11 | agents, central components | x86_64, aarch64 | | Debian | 12 | agents, central components | x86_64, aarch64 | | Ubuntu | 18 | agents | x86_64, aarch64 | | Ubuntu | 20 | agents, central components | x86_64, aarch64 | | Ubuntu | 22 | agents, central components | x86_64, aarch64 | | Oracle Linux | 9 | agents, central components | x86_64, aarch64 | | Amazon Linux | 2 | agents, central components | x86_64, aarch64 | | Amazon Linux | 2023 | agents, central components | x86_64, aarch64 | | openSUSE | 15 | agents, central components | x86_64, aarch64 | | SUSE | 15 | agents, central components | x86_64, aarch64 | | Fedora | 37 | agents | x86_64, aarch64 | | Fedora | 38 | agents | x86_64, aarch64 | | Windows | 10 | agents | x86_64, aarch64 | | Windows | 11 | agents | x86_64, aarch64 | | Windows | Server 2012 | agents | x86_64, aarch64 | | Windows | Server 2012 R2 | agents | x86_64, aarch64 | | Windows | Server 2016 | agents | x86_64, aarch64 | | Windows | Server 2019 | agents | x86_64, aarch64 | | Windows | Server 2022 | agents | x86_64, aarch64 | | macOS | Ventura | agents | x86_64, aarch64 | | macOS | Sonoma | agents | x86_64, aarch64 |

Upgrade Process

• DTT2 includes the following phases for the upgrade process:
  • Pre-upgrade checks
  • Upgrade execution
  • Post-upgrade validation

Phase	Requirement
Pre-upgrade checks	Ensure all services are running and configurations are intact before upgrade
Upgrade execution	Execute the upgrade process using the recommended method for each OS and component
Upgrade execution	Verify that the upgrade does not affect running services or processes
Post-upgrade validation	Ensure file permissions are maintained post-upgrade
Post-upgrade validation	Ensure configuration is maintained post-upgrade (ossec.conf, agent.conf, local_internal_options.conf)
Post-upgrade validation	Verify successful reconnection of agents to the manager post-upgrade
Post-upgrade validation	Confirm all components and services are operational post-upgrade

Automation

• DTT2 tests must be integrated into the Nightly CI pipeline
• DTT2 tests must be integrated into the Weekly CI pipeline
• DTT2 tests must be part of pre-release testing
• The cost and time of DTT2 tests must be measurable and optimized

Non-functional requirements

• All DTT2 test phases must comply with the following requirements:
  • Ensure the upgrade process does not exceed the maximum time defined for the specific phase
  • Ensure no errors are found in logs post-upgrade
• DTT2 tests must be designed with a modular approach for easy deployment, execution, and result analysis
• DTT2 test executions must be monitored and reviewed regularly by the QA team
• DTT2 tests evaluation criteria must be clearly defined and communicated to all QA team members
• An escalation process for DTT2 test failures must be established and documented

Hardware

Upgrade specifications

- Agents and Central Components: - Upgrade from the previous patch version - Upgrade from the previous minor version

Implementation restrictions

• DTT2 CI architecture and infrastructure must be implemented in Jenkins.
• DTT2 tests must be developed using Python.
• DTT2 must utilize virtual machines for deploying the OSs.

Plan

Iteration 2:

Objective:

Focus on resolving the upgrade-specific challenges identified in tier 1 testing. This includes ensuring that the upgrade process is robust, configurations are maintained, and the system remains secure post-upgrade.

Tasks:

• [ ] Implement automated upgrade tests for all tier 1 OSs and architectures
• [ ] Validate upgrade process integrity and consistency across all supported platforms
• [ ] Ensure all upgrade tests are integrated into the CI pipeline for regular execution

Expected Results:

• A comprehensive report on the upgrade testing across all tier 1 operating systems and architectures, including any issues identified and recommendations for improvements.

Approved by

DRI name: @davidjiglesias CTO: @havidarou Objective: Ensure robust and reliable upgrade processes for tier 1 operating systems and architectures

[rauldpm]

Related https://github.com/wazuh/wazuh-qa/issues/5137