Investigate and document tests - Check Files

[Deblintrake09]

Parent issue	https://github.com/wazuh/wazuh-qa/issues/4241
Pipeline state	:yellow_circle:
Tests state	:yellow_circle:
Jenkins link
Branch used	4.4

Description

This issue aims to analyze and document all about the Check Files test suite, located in wazuh-qa/tests/check_files. Also Jenkins automation state, Jenkins parameters, pytest location, steps in the test and current state of it.

The wazuh-jenkins repository contains a different version of the Check Files located in https://github.com/wazuh/wazuh-jenkins/blob/master/quality/tests/generic/common_files/check_files.py. This test will also be researched as part of this issue.

State of check_files in wazuh-qa :red_circle:

• The test execution was not automated and has no current pipeline. They have to be launched locally an manually.
• The tests use the qa-ctl tool that is currently deprecated and not working. It could be adapted automated using the current Deployer pipeline.
• When executing the tests check_files script did not finish.
• For more details, check comment below.

State of check_files in wazuh-jenkins :green_circle:

• The test is automated and executed as part of multiple pipelines (test_install, test_upgrade, test_service, pr_wazuh, nightly).
• Tests are currently passing as of 4.4.4-RC2.
• For more details, check comment below.

[Deblintrake09]

Research on Check Files in wazuh-qa

What is being tested

The tests located in the Wazuh-qa repository check the files in the system and the difference of files after a given install, update or uninstall process.

State

• This suite used the now deprecated qa-ctl tool which is not currently working, to set up an environment and launch the tests and gather the reports.
• The tests can be launched manually, but the process is not properly documented.

How to run tests

To check Installation files:

• Execute check_files before installation. Creates file for *--before_file** test parameter

  cd wazuh-qa
  python3 deps/wazuh_testing/wazuh_testing/scripts/check_files.py  <PARAMETERS>

• Install manager/agent
• Execute check_files again to get a second output_file. Creates file for *--after_file** test parameter
• Execute the test file in test/check_files

To check file changes during upgrade:

• Execute check_files with installed manager/agent. Creates file for *--before_file** test parameter
• Upgrade the manager/agent
• Execute check_files again to get a second output_file. Creates file for *--after_file** test parameter
• Execute the test file in test/check_files

To check uninstallation files:

• Execute check_files with installed manager/agent. Creates file for *--before_file** test parameter
• Uninstall the manager/agent
• Execute check_files again to get a second output_file. Creates file for *--after_file** test parameter
• Execute the test file in test/check_files

Parameters for check_files script

• PATH: Path base to inspect files recursively from. All files and folders in the given path will be checked. Arg: "--path", default='/'
• IGNORE: List of paths to ignore- Arg: "-i"
• OUTPUT_FILE: path to store the results. ARG: "-o" or "--output-file"
• DEBUG: Run in debug mode. Arg: '-d' or '--debug'

Parameters for the test

• BEFORE_FILE: Python Arg. --before-file. Path to check_file.py report generated before install/upgrade/uninstall.
• AFTER_FILE: Python Arg. --after-file. Path to check_file.py report generated after install/upgrade/uninstall.

NOTES: Both parameters are required as this is the data used by the test to validate the changes.

Local testing result :red_circle:

Testing could not be completed:

• When trying to launch the check_files script on a local environment, for the pre-installation check, the script was left to work for 6 hours and it did not finish.
• During research, the script has been said it would take time depending on what is in the system. The testing was done on a clean Ubuntu22 VM. 6 hours would be unacceptable and is not the expected time it would take.

Testing results

- Check wazuh is not installed ``` # ls /var/ossec ls: cannot access '/var/ossec': No such file or directory ``` - Launch check_files.py ``` # cd /vagrant/wazuh-qa/deps/wazuh_testing/wazuh_testing/scripts/ #python3 check_files.py -o /vagrant/pre-install.json -d 2023-07-04 16:24:30,904 - INFO - Ignoring the following paths: [] 2023-07-04 16:24:30,904 - INFO - Getting check-files data from / ```

[Deblintrake09]

Research on Check Files in wazuh-jenkins

What is tested

This check_files script is a Python script that checks that with a given installed version of wazuh, the expected file structure and permissions are found.

The script uses a series of JSON files that list the expected files, permissions in the different versions of wazuh.

• The complete documentation is located here.

State :green_circle:

• Currently, these tests are passing on all OSs for 4.4.4 as checked in Release Candidate 2.
• Pipeline Example: https://ci.wazuh.info/view/Tests/job/Test_install/129733

  16:57:01  [Pipeline] ansiblePlaybook
  16:57:01  [Test_install@2] $ ansible-playbook quality/tests/generic/02-installation/linux/test/conf/test_agent.yaml -i solaris_vagrant_ansible_host -l Agents --private-key /home/ec2-user/workspace/Test_install@2/ssh2845410060375437077.key -u root -e curl_path=/opt/csw/bin/curl -e logs_folder=/home/ec2-user/workspace/Test_install@2/artifacts -e wazuh_base_version=master -e to_version=4.7.0 -e from_version=4.7.0 -e python_path=/usr/testing/bin/python3 -e pytest_path=/usr/testing/bin/pytest -e pytest_args=--tb=short
  16:57:02  [WARNING]: Found both group and host with the same name: Agents
  16:57:04  
  16:57:04  PLAY [Agents] ******************************************************************
  16:57:09  
  16:57:09  TASK [Gathering Facts] *********************************************************

Usage status

• This script is used as part of the following test playbooks:

  • As part of the Upgrade Test suite, it is referenced in the following files:
    • wazuh-jenkins/quality/tests/generic/07-upgrade-wpk/linux/test/conf/test_agent.yaml
    • wazuh-jenkins/quality/tests/generic/03-upgrade/linux/test/conf/test_manager.yaml
    • wazuh-jenkins/quality/tests/generic/03-upgrade/linux/test/conf/test_agent.yaml
  • As part of the installation test suite, it is referenced in the following files, but it is not used:
    • wazuh-jenkins/quality/tests/generic/02-installation/linux/test/conf/test_agent.yaml
    • wazuh-jenkins/quality/tests/generic/02-installation/linux/test/conf/test_manager.yaml

• In terms of Pipelines it is used in:

  • Test_install - only referenced, not launched or used
  • Test_upgrade - tests are executed in this pipeline
  • PR_wazuh - launched by test_upgrade as part of the pipeline but not used by the pipeline itself
  • test_nightly - launched by test_upgrade as part of the pipeline but not used by the pipeline itself

Used files

• check_files_data.json: Data file containing all check file configurations (similar to database).
• linux_upgrade_check_files: Template file containing all the check_files (by type of installation and operating system) necessary to check in agent, manager, and API (in this case upgrade Linux). The idea is to make a template for each operating system different and each type of test (installation, upgrade, uninstall ...)
• check_files_exceptions: File to store all the files that are not included in check files data and want to be ignored.
• check_files.py: Executable file that loads data and check_files templates and checks them according to a series of input parameters such as target, operating system...

JSON data structure format

• When files need to be added to one of the data files, the information needed needs to follow this example:

  {
  "data": [
      {
          "id": 0,
          "name": "/var/ossec",
          "description": {
              "class": "static",
              "group": "ossec",
              "mode": "0750",
              "prot": "drwxr-x---",
              "type": "directory",
              "user": "root"
          }
      },
      {
          "id": 1,
          "name": "/var/ossec/.ssh",
          "description": {
              "class": "static",
              "group": "ossec",
              "mode": "0700",
              "prot": "drwx------",
              "type": "directory",
              "user": "root"
          }
      },
  ]
  }

• In case needing to add, modify or delete files from a structure, the changes can apply to the corresponding file, ensuring the JSON format is not broken.

Failure example:

When failing, the error will state what was expected and what was found, displayed in the following way:

17:22:07
17:22:07 Different:
17:22:07 /var/ossec/.ssh [Wrong: mode]
17:22:07 Expected: root ossec 0700 # drwx------
17:22:07 Found: root ossec 0750 # drwxr-x---
17:22:07

17:22:07 Please review your files.

Parameters

• TARGET: Check files for manager or agent. Arg: "-t" or "--target", choices=['manager', 'agent']
• TEST: Type of test. Arg: "-a", "--test", choices=['upgrade', 'uninstall']
• VERSION: Version that is being checked. Arg: "-v", "--version". Ex: 4.4.4
• OS: Operating system or host distribution. Arg: "-o", "--os". choices=['linux', 'windows', 'redhat', 'debian']
• IGNORE: Path to ignore. Arg: "-i", "--ignore", Example: /var/ossec/wodles/oscap/content,/var/ossec/api
• NO_SHOW_IGNORE_LIST: If true Ignored files list is not shown. Arg: "-n", "--no_show_ignore_list"
• DIRECTORY: Used to change the Installation location. Arg: "-d", "--directory"
• FROM_VERSION: Version you come from, to exclude files of it. Arg: "-f", "--from_version"

[rauldpm]

Review notes

• Check grammar and nomenclature (JSON, wazuh-qa, etc)
• I think that is needed more documentation about:
  • Data structure examples
  • Used files
  • Usage examples
  • How to add things to the structure
  • The test output when the check-files test fails,
• You can reuse the documentation already created about wazuh-jenkins check-files: https://github.com/wazuh/wazuh-jenkins/blob/master/doc/source/jenkins/ci/checkfile_test.rst
• On the other hand, shouldn't this knowledge be reflected in the Google Sites documentation?

[Deblintrake09]

Applied requested changes

[rauldpm]

Review notes

wazuh-jenkins

Pipeline Example: https://ci.wazuh.info/view/Tests/job/Test_install/129733

The Test_install pipelines do not use the check-files.py script nor check the file's properties, they are used only for upgrade tests

As part of Installation test suite: wazuh-jenkins/quality/tests/generic/02-installation/linux/test/conf/test_agent.yaml wazuh-jenkins/quality/tests/generic/02-installation/linux/test/conf/test_manager.yaml

Yes, the Python script is referenced but it is not used in the playbooks

In terms of Pipelines it is used in:

• PR_Wazuh
• Nightly
• Test_install
• Test_upgrade
• Test_service

Where is the Test_service pipeline using the check-files?

---

Shouldn't this knowledge be reflected in the Google Sites documentation?

[Deblintrake09]

• Updated test_install to note it is only referenced, but not used.
• Removed test_service from list.
• This Epic's scope does not include the generation of documentation on Google Sites, as this was a research stage and documentation will be left in the related issues. After the results are checked and all tests are determined to be used or deprecated/deleted, a documentation phase will be started to create documentation for what is left.

[rauldpm]

Review notes

Updated test_install to note it is only referenced, but not used.

It must be clear that the Test_install pipeline does not use the check-files

• As part of the installation test suite, it is referenced in the following files but they are not used.

---

In terms of Pipelines it is used in: Test_install - only referenced but not actually used. Test_upgrade - tests are actually executed in this pipeline PR_Wazuh Nightly

Add a note in PR_wazuh and test_nightly stating that the check-files are launched in the Tests_ugrade pipeline and not in the PR_wazuh or test_nightly pipelines. Also, refer to the pipelines with their actual names (Nightly -> test_nightly)

---

Please check the grammar of the entire comment

VERSION: Vversion of Installation this (these) tests are passing

---

• I don't have enough knowledge about wazuh-qa to validate the comment about wazuh-qa

[Deblintrake09]

Applied requested changes.

[rauldpm]

Review notes

• Approved changes, I request a second review about the wazuh-qa section

[mauromalara]

Review comment No.5

Great research and description 👏🏻 👏🏻

Only a few changes are required:

Check_files on wazuh-qa

What is test

Change the title to “What is being tested”

How to run tests

You mention this in “What is tested”:

[the tests] check the files in the system and the difference of files after a given install, update or uninstall process.

But then you only explain how to check files before/after an install process. Please, add more detail to understand how the tool is used in all the cases.

Parameters for check_files script

PATH: Path base to inspect files recursively. Arg: "--path", default='/'

I don’t quite understand what is this parameter about

Parameters for

I think the title is incomplete, am I right?

BEFORE_FILE: Python Arg. --before-file. Path to check_file.py report before install/upgrade/uninstall.

I would change this “Path to check_file.py report…” By this “Path where the script will report….”

AFTER_FILE: Python Arg. --after-file. Path to check_file.py report after install/upgrade/uninstall.

IDEM above.

[Deblintrake09]

Response to review Comment No. 5

• Changed title.
• The steps for upgrade or uninstall were the same for installation. Have updated explicit commands for each type.
• Updated PATH parameter description. This path is the base path that files fill be checked. If you use the default for example all files and folders starting from / will be checked. If the path is changed to /var/ossec only files and folders inside that path would be checked.
• Updated the title for the parameters for the test.
• BEFORE_FILE and AFTER_FILE are the paths to the files created by the script before, not where the test will report. These files should exist before launching the tests.

[mauromalara]

Review comment No.6

BEFORE_FILE and AFTER_FILE are the paths to the files created by the script before, not where the test will report. These files should exist before launching the tests.

I'm still seeing the previous description, you explained it well in the previous comment. Please, expand the following descriptions for everyone to understand:

BEFORE_FILE: Python Arg. --before-file. Path to check_file.py report before install/upgrade/uninstall. AFTER_FILE: Python Arg. --after-file. Path to check_file.py report after install/upgrade/uninstall.

Note: Please, check the QA project's fields, especially the ETA cannot be empty.

[mauromalara]

LGTM

[davidjiglesias]

LGTM!