search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
61 stars 30 forks source link

Upgrade `setuptools` dependency #3760

Closed jmv74211 closed 3 months ago

jmv74211 commented 1 year ago

It is necessary to upgrade the setuptools dependency to a version > 65.5.1.

Note: Will be necessary to check if the dependency upgrade has any bad impact when installing the wazuh-qa framework in any system.

Rebits commented 1 year ago

Update 17/01/2023

Development Branch
3760-setuptools-dep

Testing framework installation

OS Ubuntu Windows Server centOS Solaris macOS
Python Version 3.6 [:red_circle:](https://ci.wazuh.info/job/Test_integration/35596/console) [:red_circle:](https://ci.wazuh.info/job/Test_integration/35596/console) [:red_circle:](https://ci.wazuh.info/job/Test_integration/35596/console) :black_circle: :black_circle:
3.10 :green_circle: :green_circle: :green_circle: NA NA

Evidence

Python3.10 :green_circle:
Ubuntu ``` Successfully installed charset-normalizer-3.0.1 prettytable-3.6.0 pytest-7.1.2 requests-2.28.2 setuptools-66.0.0 tomli-2.0.1 urllib3-1.26.14 wcwidth-0.2.6 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available. You should consider upgrading via the '/usr/local/bin/python3.10 -m pip install --upgrade pip' command. root@ip-172-31-1-53:/home/qa/wazuh-qa# ```
Debian ``` Requirement already satisfied: mypy-extensions>=0.3.0 in /usr/local/lib/python3.10/site-packages (from typing-inspect>=0.4.0->libcst>=0.3.10->google-cloud-pubsub==2.3.0->-r requirements.txt (line 10)) (0.4.3) Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.25.0->google-api-core[grpc]<2.0.0dev,>=1.22.2->google-cloud-pubsub==2.3.0->-r requirements.txt (line 10)) (0.4.8) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: There was an error checking the latest version of pip. ```
centOS ``` Successfully installed charset-normalizer-3.0.1 prettytable-3.6.0 pytest-7.1.2 requests-2.28.2 setuptools-66.0.0 tomli-2.0.1 urllib3-1.26.14 wcwidth-0.2.6 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv WARNING: There was an error checking the latest version of pip. ```
RHEL ``` Successfully installed charset-normalizer-3.0.1 prettytable-3.6.0 pytest-7.1.2 requests-2.28.2 setuptools-66.0.0 tomli-2.0.1 urllib3-1.26.14 wcwidth-0.2.6 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [root@ip-172-31-6-63 wazuh-qa]# uname -a Linux ip-172-31-6-63.ec2.internal 5.10.126-117.518.amzn2.x86_64 #1 SMP Wed Jun 29 23:49:26 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [root@ip-172-31-6-63 wazuh-qa]# ```
Windows Server ``` Successfully installed charset-normalizer-3.0.1 docker-6.0.1 pytest-7.1.2 requests-2.28.2 setuptools-66.0.0 tomli-2.0.1 urllib3-1.26.14 websocket-client-1.4.2 WARNING: You are using pip version 22.0.4; however, version 22.3.1 is available. You should consider upgrading via the 'C:\Users\qa\AppData\Local\Programs\Python\Python310\python.exe -m pip install --upgrade pip' command. ```
Python 3.6 :red_circle:
It is not possible to use version `65.5.1` of setuptools for python 3.6 ``` 16:32:06 Downloading scipy-1.5.4-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB) 16:32:06 Collecting seaborn>=0.11.1 16:32:06 Downloading seaborn-0.11.2-py3-none-any.whl (292 kB) 16:32:06 16:32:06 16:32:06 STDERR: 16:32:06 16:32:06 ERROR: Could not find a version that satisfies the requirement setuptools>65.5.1 (from versions: 0.6b1, 0.6b2, 0.6b3, 0.6b4, 0.6rc1, 0.6rc2, 0.6rc3, 0.6rc4, 0.6rc5, 0.6rc6, 0.6rc7, 0.6rc8, 0.6rc9, 0.6rc10, 0.6rc11, 0.7.2, 0.7.3, 0.7.4, 0.7.5, 0.7.6, 0.7.7, 0.7.8, 0.8, 0.9, 0.9.1, 0.9.2, 0.9.3, 0.9.4, 0.9.5, 0.9.6, 0.9.7, 0.9.8, 1.0, 1.1, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.2, 1.3, 1.3.1, 1.3.2, 1.4, 1.4.1, 1.4.2, 2.0, 2.0.1, 2.0.2, 2.1, 2.1.1, 2.1.2, 2.2, 3.0, 3.0.1, 3.0.2, 3.1, 3.2, 3.3, 3.4, 3.4.1, 3.4.2, 3.4.3, 3.4.4, 3.5, 3.5.1, 3.5.2, 3.6, 3.7, 3.7.1, 3.8, 3.8.1, 4.0, 4.0.1, 5.0, 5.0.1, 5.0.2, 5.1, 5.2, 5.3, 5.4, 5.4.1, 5.4.2, 5.5, 5.5.1, 5.6, 5.7, 5.8, 6.0.1, 6.0.2, 6.1, 7.0, 8.0, 8.0.1, 8.0.2, 8.0.3, 8.0.4, 8.1, 8.2, 8.2.1, 8.3, 9.0, 9.0.1, 9.1, 10.0, 10.0.1, 10.1, 10.2, 10.2.1, 11.0, 11.1, 11.2, 11.3, 11.3.1, 12.0, 12.0.1, 12.0.2, 12.0.3, 12.0.4, 12.0.5, 12.1, 12.2, 12.3, 12.4, 13.0.1, 13.0.2, 14.0, 14.1, 14.1.1, 14.2, 14.3, 14.3.1, 15.0, 15.1, 15.2, 16.0, 17.0, 17.1, 17.1.1, 18.0, 18.0.1, 18.1, 18.2, 18.3, 18.3.1, 18.3.2, 18.4, 18.5, 18.6, 18.6.1, 18.7, 18.7.1, 18.8, 18.8.1, 19.0, 19.1, 19.1.1, 19.2, 19.3, 19.4, 19.4.1, 19.5, 19.6, 19.6.1, 19.6.2, 19.7, 20.0, 20.1, 20.1.1, 20.2.2, 20.3, 20.3.1, 20.4, 20.6.6, 20.6.7, 20.6.8, 20.7.0, 20.8.0, 20.8.1, 20.9.0, 20.10.1, 21.0.0, 21.1.0, 21.2.0, 21.2.1, 21.2.2, 22.0.0, 22.0.1, 22.0.2, 22.0.4, 22.0.5, 23.0.0, 23.1.0, 23.2.0, 23.2.1, 24.0.0, 24.0.1, 24.0.2, 24.0.3, 24.1.0, 24.1.1, 24.2.0, 24.2.1, 24.3.0, 24.3.1, 25.0.0, 25.0.1, 25.0.2, 25.1.0, 25.1.1, 25.1.2, 25.1.3, 25.1.4, 25.1.5, 25.1.6, 25.2.0, 25.3.0, 25.4.0, 26.0.0, 26.1.0, 26.1.1, 27.0.0, 27.1.0, 27.1.2, 27.2.0, 27.3.0, 27.3.1, 28.0.0, 28.1.0, 28.2.0, 28.3.0, 28.4.0, 28.5.0, 28.6.0, 28.6.1, 28.7.0, 28.7.1, 28.8.0, 28.8.1, 29.0.0, 29.0.1, 30.0.0, 30.1.0, 30.2.0, 30.2.1, 30.3.0, 30.4.0, 31.0.0, 31.0.1, 32.0.0, 32.1.0, 32.1.1, 32.1.2, 32.1.3, 32.2.0, 32.3.0, 32.3.1, 33.1.0, 33.1.1, 34.0.0, 34.0.1, 34.0.2, 34.0.3, 34.1.0, 34.1.1, 34.2.0, 34.3.0, 34.3.1, 34.3.2, 34.3.3, 34.4.0, 34.4.1, 35.0.0, 35.0.1, 35.0.2, 36.0.1, 36.1.0, 36.1.1, 36.2.0, 36.2.1, 36.2.2, 36.2.3, 36.2.4, 36.2.5, 36.2.6, 36.2.7, 36.3.0, 36.4.0, 36.5.0, 36.6.0, 36.6.1, 36.7.0, 36.7.1, 36.7.2, 36.8.0, 37.0.0, 38.0.0, 38.1.0, 38.2.0, 38.2.1, 38.2.3, 38.2.4, 38.2.5, 38.3.0, 38.4.0, 38.4.1, 38.5.0, 38.5.1, 38.5.2, 38.6.0, 38.6.1, 38.7.0, 39.0.0, 39.0.1, 39.1.0, 39.2.0, 40.0.0, 40.1.0, 40.1.1, 40.2.0, 40.3.0, 40.4.0, 40.4.1, 40.4.2, 40.4.3, 40.5.0, 40.6.0, 40.6.1, 40.6.2, 40.6.3, 40.7.0, 40.7.1, 40.7.2, 40.7.3, 40.8.0, 40.9.0, 41.0.0, 41.0.1, 41.1.0, 41.2.0, 41.3.0, 41.4.0, 41.5.0, 41.5.1, 41.6.0, 42.0.0, 42.0.1, 42.0.2, 43.0.0, 44.0.0, 44.1.0, 44.1.1, 45.0.0, 45.1.0, 45.2.0, 45.3.0, 46.0.0, 46.1.0, 46.1.1, 46.1.2, 46.1.3, 46.2.0, 46.3.0, 46.3.1, 46.4.0, 47.0.0, 47.1.0, 47.1.1, 47.2.0, 47.3.0, 47.3.1, 47.3.2, 48.0.0, 49.0.0, 49.0.1, 49.1.0, 49.1.1, 49.1.2, 49.1.3, 49.2.0, 49.2.1, 49.3.0, 49.3.1, 49.3.2, 49.4.0, 49.5.0, 49.6.0, 50.0.0, 50.0.1, 50.0.2, 50.0.3, 50.1.0, 50.2.0, 50.3.0, 50.3.1, 50.3.2, 51.0.0, 51.1.0, 51.1.0.post20201221, 51.1.1, 51.1.2, 51.2.0, 51.3.0, 51.3.1, 51.3.2, 51.3.3, 52.0.0, 53.0.0, 53.1.0, 54.0.0, 54.1.0, 54.1.1, 54.1.2, 54.1.3, 54.2.0, 56.0.0, 56.1.0, 56.2.0, 57.0.0, 57.1.0, 57.2.0, 57.3.0, 57.4.0, 57.5.0, 58.0.0, 58.0.1, 58.0.2, 58.0.3, 58.0.4, 58.1.0, 58.2.0, 58.3.0, 58.4.0, 58.5.0, 58.5.1, 58.5.2, 58.5.3, 59.0.1, 59.1.0, 59.1.1, 59.2.0, 59.3.0, 59.4.0, 59.5.0, 59.6.0) 16:32:06 ERROR: No matching distribution found for setuptools>65.5.1 ``` It is required to upgrade IT instances to allow an increase in the version of the dependency.

Normal markdown text

Note Evidence was taken manually python3.10 instances

Setuptools upgrade will be blocked until python upgrade in the IT instances https://github.com/wazuh/wazuh-qa/issues/3770

Rebits commented 1 year ago

Update 20/01/2023

The proposed setuptools version was setuptools>65.5.1, installing in Ubuntu22 agents the last 66.1.0 version. This change produces pkg_resources.extern.packaging.version.InvalidVersion

Error details ``` /usr/local/lib/python3.10/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools. warnings.warn( /usr/local/lib/python3.10/dist-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools. warnings.warn( Traceback (most recent call last): File "/tmp/Test_integration_B35820_20230120133553/deps/wazuh_testing/setup.py", line 3, in setup() File "/usr/local/lib/python3.10/dist-packages/setuptools/__init__.py", line 108, in setup return distutils.core.setup(**attrs) File "/usr/lib/python3.10/distutils/core.py", line 148, in setup dist.run_commands() File "/usr/lib/python3.10/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/usr/local/lib/python3.10/dist-packages/setuptools/dist.py", line 1213, in run_command super().run_command(command) File "/usr/lib/python3.10/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/usr/local/lib/python3.10/dist-packages/setuptools/command/install.py", line 74, in run self.do_egg_install() File "/usr/local/lib/python3.10/dist-packages/setuptools/command/install.py", line 117, in do_egg_install cmd.ensure_finalized() # finalize before bdist_egg munges install cmd File "/usr/lib/python3.10/distutils/cmd.py", line 107, in ensure_finalized self.finalize_options() File "/usr/local/lib/python3.10/dist-packages/setuptools/command/easy_install.py", line 311, in finalize_options self.local_index = Environment(self.shadow_path + sys.path) File "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py", line 1044, in __init__ self.scan(search_path) File "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py", line 1077, in scan self.add(dist) File "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py", line 1096, in add dists.sort(key=operator.attrgetter('hashcmp'), reverse=True) File "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py", line 2631, in hashcmp self.parsed_version, File "/usr/local/lib/python3.10/dist-packages/pkg_resources/__init__.py", line 2685, in parsed_version raise packaging.version.InvalidVersion(f"{str(ex)} {info}") from None pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '1.1build1' (package: distro-info) ```

This is a bug in which python3-distro-info does not fit with the PEP 440-compliant versions, making the parser fails. It is proposed to use the setuptools=65.6.0 to avoid it.

Build: https://ci.wazuh.info/job/Test_integration/35823/console

Python310 installed in the instances fail importing SSL

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/ssl.py", line 98, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'

This was solved following these steps:

  156  wget https://www.openssl.org/source/openssl-1.1.1g.tar.gz
  157  tar -zxf openssl-1.1.1g.tar.gz && cd openssl-1.1.1g
  158  ./config
  159  ./config --prefix=/usr/local/custom-openssl --openssldir=/etc/ssl
  160  make -j1 depend
  161  make -j8
  162  make install_sw
  166  cd Python-3.10.9
  167  ./configure -C --with-openssl=/usr/local/custom-openssl --with-openssl-rpath=auto --prefix=/usr/local/python-3.version
  168  make -j8
  169  make alt install
  207  /usr/local/python-3.version/bin/python3.10
Rebits commented 1 year ago

Update 14/02/2023

Regarding this error, it seems that it is fixed after upgrading windows' python to 3.10. It is required to reopen https://github.com/wazuh/wazuh-qa/issues/3770 and upgrade also windows agent.

17:04:13 DEPRECATION: progress is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

17:04:13 DEPRECATION: docopt is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

17:04:13 DEPRECATION: ordered-set is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

17:04:13 DEPRECATION: future is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

17:04:13 DEPRECATION: configobj is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

17:04:13 DEPRECATION: treelib is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

17:04:13 DEPRECATION: pybitbucket-fork is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559


Regarding this error, we should try to integrate the `--use-pep517`  option in the setup script
- Packages would be ignored

17:01:15 ############################ 17:01:15 # Package would be ignored # 17:01:15 ############################ 17:01:15 Python recognizes 'wazuh_testing.data' as an importable package, 17:01:15 but it is not listed in the packages configuration of setuptools. 17:01:15
17:01:15 'wazuh_testing.data' has been automatically added to the distribution only 17:01:15 because it may contain data files, but this behavior is likely to change 17:01:15 in future versions of setuptools (and therefore is considered deprecated). 17:01:15
17:01:15 Please make sure that 'wazuh_testing.data' is included as a package by using 17:01:15 the packages configuration field or the proper discovery methods 17:01:15 (for example by using find_namespace_packages(...)/find_namespace: 17:01:15 instead of find_packages(...)/find:). 17:01:15
17:01:15 You can read more about "package discovery" and "data files" on setuptools 17:01:15 documentation page. 17:01:15 



Regarding this error, it seems that since PEP 420, there is no way for differentiating a folder from a package in Python. So, directories of the repository are considered as namespace packages. It should be necessary to replace the `find_packages`
of the setup script with `find_namespace_packages`

**Build**: https://ci.wazuh.info/job/Test_integration/36708/console
Rebits commented 12 months ago

23/05/2023

Deprecation warnings in the windows endpoint 

C:\Python37\lib\site-packages\setuptools\command\install.py:37: SetuptoolsDeprecationWarning: setup.py install is deprec
ated. Use build and pip and other standards-based tools.
  setuptools.SetuptoolsDeprecationWarning,
C:\Python37\lib\site-packages\setuptools\command\easy_install.py:147: EasyInstallDeprecationWarning: easy_install comman
d is deprecated. Use build and pip and other standards-based tools.

In order to avoid produced errors and warnings is required to migrate to pyproject.toml schema. Further research is required

Rebits commented 10 months ago

28/06/2023

Rebits commented 10 months ago

29/06/2023

14:24:23 File "C:\Python311\Lib\site-packages\py_vendored_packages\apipkg__init.py", line 145, in makeattr

14:24:23 modpath, attrname = self.map[name]

14:24:23 ~~~~^^^^^^

14:24:23 KeyError: 'spec'


It seems that error is fixed when py is upgraded to 1.11.0 

***

- Solved installation errors
- Open Wazuh Jenkins PR to upgrade Windows IT ami and change setup.py deprecated installation method (https://github.com/wazuh/wazuh-jenkins/pull/5342)
- Testing IT pipeline using logcollector module: https://ci.wazuh.info/view/Tests/job/Test_integration/40719/
Rebits commented 10 months ago

03/07/2023

Rebits commented 10 months ago
jnasselle commented 7 months ago

Reopening this issue and rescheduling it to 4.8.0 to propper implementation among other wazuh-qa repo consumers like wazuh-jenkins

pro-akim commented 7 months ago

Update

Understanding that the setuptool has security issues (mentioned in https://github.com/wazuh/wazuh/issues/16473)

Researching https://scout.docker.com/vulnerabilities/id/CVE-2022-40897

CVE-2022-40897 SOURCE - github Summary

Python Packaging Authority (PyPA)'s setuptools is a library designed to facilitate packaging Python projects. Setuptools version 65.5.0 and earlier could allow remote attackers to cause a denial of service by fetching malicious HTML from a PyPI package or custom PackageIndex page due to a vulnerable Regular Expression in package_index. This has been patched in version 65.5.1.

On the other hand: https://github.com/wazuh/wazuh-jenkins/issues/5695 shows that the changes done in wazuh-qa affected functionalities of other pipelines/repositories requiring a deeper understanding of its influence.

It seems that test_service pipeline is affected in all its options of test.

requirements.txt use

It could potentially exist in the following files where the requirement.txt file is utilized. Therefore, if there is a change in setuptools, all tests, pipelines, or deployments involving these files may fail.

pro-akim commented 7 months ago

Update

On the other hand, setuptools > 65.5.1 requires Python >=3.7

This would imply modifying the Python version of all products that consume requirements.txt (many of them are working on Python 3.6). It will be important to weigh whether these products are deliverables and if they can be exposed to an attack, versus the work involved in retesting all the sectors that use that list of libraries after the update.

Checking the current version of Python in each of the products where requirements.txt is consumed:

file python-vers
doc>Dockerfile 3.7-alpine
dockers>repositories>wazuh>api_performance>Dockerfile 3.9
dockers>repositories>wazuh-jenkins>data_visualization>Dockerfile 3.9
dockers>repositories>wazuh-jenkins>log_parser>Dockerfile 3.9
dockers>repositoriesreport_generator>Dockerfile 3.9-buster
dockers>repositories>wazuh>simulate_agents>Dockerfile 3.9
quality>tests>performance>fim>roles>test-fim>tasks>configure_agent.yaml 3.7?
dockers>repositories>wazuh>api_performance>run_api_performance_test.sh NA
dockers>repositories>wazuh>simulate_agents>run_simulated_agents.sh NA
dockers>repositories>wazuh>simulate_api>Dockerfile NA
dockers>repositories>wazuh>simulate_api>run_api_simulation.sh NA
dockers>repositories>wazuh-jenkins>aws_docker_instance>Dockerfile NA
dockers>repositories>wazuh-jenkins>data_visualization>data_visualizacion.sh NA
dockers>repositories>wazuh-jenkins>log_parser>log_parser.sh NA
dockers>repositories>wazuh-jenkins>report_generator>report_generator.sh NA
jenkins-files>tests>test_e2e_system.groovy NA
jenkins-files>tests>test_integration.groovy NA
quality>tests>generic>08-service>linux>test>conf>test-manager>tasks>main.yaml NA
quality>tests>generic>08-service>linux>test>conf>test-agent>tasks>main.yaml NA
quality>tests>performance>cluster>cluster_tests.sh NA
quality>tests>cluster>manager.provision.yaml NA
quality>tests>performance>fim>roles>test-fim>tasks>configure_manager.yaml NA
quality>tests>performance>fim>roles>test-fim>tasks>configure_win_agent.yaml NA
quality>tests>reliability>agent_provision.yaml NA
quality>tests>reliability>manager_privision.yaml NA
quality>tests>reliability>test_reliability.yaml NA
quality>tests>system>playbooks>provision.yaml NA

where:

are involved in test_service.

The update of Python in test_service is being evaluated initially.

pro-akim commented 7 months ago

Update

When attempting to change the version of setuptools, it was necessary to install Python version 3.7 or higher. During this installation, messages related to other libraries conflicting with the new version appeared, and it also became challenging to set the path for pip3 and Python.

pro-akim commented 6 months ago

Update

The decision is made to work locally to avoid conflicts between existing libraries and the Python version to be installed. Afterwards, the same will be utilized in an ECR/AMI, where tests will be executed, thus simplifying provisioning and eliminating ambiguous steps in each of the tests. Therefore, progress is made in the research of the ideal Python version, ranging from LTS to older versions.

Following steps will be followed:

  1. Deploy the image/ECR with the Python to be updated.
  2. Update Python.
  3. Local tests -> (Framework can be installed)
  4. Clean the environment of all intermediate files we created.
  5. Create a temporary AMI/ECR with updated Python.
  6. Launch pipelines or processes with the new AMI on a custom branch.
  7. From the temporary AMI, create the final AMI/ECR.
  8. Ensure that everything works as it should.
pro-akim commented 6 months ago

Update

Python version Status Conflict
3.10 :green_circle:
3.11 :red_circle: netifaces
3.12 :red_circle: git-repo
pro-akim commented 6 months ago

Update

Python version Status Conflict
3.10 :green_circle:
3.11 :red_circle: jq
3.12 :red_circle: git-subprocess
pro-akim commented 6 months ago

Update

Creating Dockerfile and testing python 3.10 install in different OS

pro-akim commented 6 months ago

Update

Issues around: 

netifaces.c:1:10: fatal error: Python.h: No such file or directory           1 | #include <Python.h>             |          ^~~~~~~~~~

Docker is not working as Vagrant environment.

pro-akim commented 6 months ago

Update

  1. :green_circle: Deploy the image/ECR with the Python to be updated.
  2. :green_circle: Update Python.
  3. :green_circle: Local tests -> (Framework can be installed)
  4. :green_circle: Clean the environment of all intermediate files we created.
  5. :black_circle: Create a temporary AMI/ECR with updated Python.
  6. :black_circle: Launch pipelines or processes with the new AMI on a custom branch.
  7. :black_circle: From the temporary AMI, create the final AMI/ECR.
  8. :black_circle: Ensure that everything works as it should.

Tested Dockerfiles for ECR:

Ubuntu:

``` FROM ubuntu:20.04 RUN apt-get update -y && apt-get install -y software-properties-common && add-apt-repository ppa:deadsnakes/ppa && apt-get update -y && apt-get install -y python3.10 python3-pip python3.10-dev libffi-dev libssl-dev RUN apt-get install -y python3.10-distutils RUN apt-get install -y curl RUN curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py RUN python3.10 get-pip.py ```

Centos:

``` FROM centos:8 RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* RUN yum -y install wget yum-utils make gcc openssl-devel bzip2-devel libffi-devel zlib-devel ARG AMB=4 RUN wget https://www.python.org/ftp/python/3.10.8/Python-3.10.8.tgz && \ tar xzf Python-3.10.8.tgz && \ cd Python-3.10.8 && \ ./configure --with-system-ffi --with-computed-gotos --enable-loadable-sqlite-extensions && \ make -j $AMB && \ make altinstall && \ pip3.10 install --upgrade pip ```
pro-akim commented 6 months ago

Update

  1. :green_circle: Deploy the image/ECR with the Python to be updated.
  2. :green_circle: Update Python.
  3. :green_circle: Local tests -> (Framework can be installed)
  4. :green_circle: Clean the environment of all intermediate files we created.
  5. :green_circle: Create a temporary AMI/ECR with updated Python.
  6. :black_circle: Launch pipelines or processes with the new AMI on a custom branch.
  7. :black_circle: From the temporary AMI, create the final AMI/ECR.
  8. :black_circle: Ensure that everything works as it should.

Repositories created in ECR. Starting testing stage

pro-akim commented 6 months ago

Update

Test_service pipeline:

pro-akim commented 6 months ago

Update

``` 11:39:41 task path: /home/ec2-user/workspace/Test_service/quality/tests/generic/08-service/linux/test/conf/test-manager/tasks/main.yaml:92 11:39:41 fatal: [Test_service_B6867_2023102693045_manager]: FAILED! => { 11:39:41 "changed": true, 11:39:41 "cmd": [ 11:39:41 "/usr/local/bin/python3.10", 11:39:41 "-m", 11:39:41 "pytest", 11:39:41 "--tb=short", 11:39:41 "-v", 11:39:41 "/wazuh-qa/tests/integration/test_service.py" 11:39:41 ], 11:39:41 "delta": "0:00:00.459414", 11:39:41 "end": "2023-10-26 09:39:41.540254", 11:39:41 "invocation": { 11:39:41 "module_args": { 11:39:41 "_raw_params": "/usr/local/bin/python3.10 -m pytest \"--tb=short\" -v /wazuh-qa/tests/integration/test_service.py", 11:39:41 "_uses_shell": false, 11:39:41 "argv": null, 11:39:41 "chdir": null, 11:39:41 "creates": null, 11:39:41 "executable": null, 11:39:41 "removes": null, 11:39:41 "stdin": null, 11:39:41 "stdin_add_newline": true, 11:39:41 "strip_empty_ends": true, 11:39:41 "warn": true 11:39:41 } 11:39:41 }, 11:39:41 "rc": 4, 11:39:41 "start": "2023-10-26 09:39:41.080840" 11:39:41 } 11:39:41 11:39:41 STDERR: 11:39:41 11:39:41 ImportError while loading conftest '/wazuh-qa/tests/integration/conftest.py'. 11:39:41 /wazuh-qa/tests/integration/conftest.py:18: in 11:39:41 from wazuh_testing import ALERTS_JSON_PATH, ARCHIVES_JSON_PATH, ARCHIVES_LOG_PATH, global_parameters, logger, mocking 11:39:41 /usr/local/lib/python3.10/site-packages/wazuh_testing/mocking/__init__.py:5: in 11:39:41 from wazuh_testing.db_interface import global_db 11:39:41 /usr/local/lib/python3.10/site-packages/wazuh_testing/db_interface/__init__.py:5: in 11:39:41 import sqlite3 11:39:41 /usr/local/lib/python3.10/sqlite3/__init__.py:57: in 11:39:41 from sqlite3.dbapi2 import * 11:39:41 /usr/local/lib/python3.10/sqlite3/dbapi2.py:27: in 11:39:41 from _sqlite3 import * 11:39:41 E ModuleNotFoundError: No module named '_sqlite3' 11:39:41 11:39:41 11:39:41 MSG: ```
pro-akim commented 6 months ago

Update

Creating CentOS 6 and 7, Ubuntu bionic and trusty AMIs with python3.10

pro-akim commented 6 months ago

Update

Some issues running CentOS 6 and 7 https://ci.wazuh.info/job/Test_service/6877/ https://ci.wazuh.info/job/Test_service/6874/

Also in Ubuntu Bionic and Trusty: https://ci.wazuh.info/job/Test_service/6875/ https://ci.wazuh.info/job/Test_service/6880/

with the following error:

``` 19:48:32 Could not fetch URL https://pypi.org/simple/configobj/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/configobj/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping 19:48:32 Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping 19:48:32 19:48:32 :stderr: WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. 19:48:32 WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/configobj/ 19:48:32 WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/configobj/ 19:48:32 WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/configobj/ 19:48:32 WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/configobj/ 19:48:32 WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/configobj/ 19:48:32 ERROR: Could not find a version that satisfies the requirement configobj==5.0.6 (from versions: none) 19:48:32 ERROR: No matching distribution found for configobj==5.0.6 19:48:32 WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. 19:48:32 WARNING: There was an error checking the latest version of pip. ```
pro-akim commented 6 months ago

Upgrade

After reviewing the AMIs: CentOS 7 was handled: manually the installation of dependencies worked however during the pipeline action https://ci.wazuh.info/job/Test_service/6887/console

``` 10:52:28 fatal: [Test_service_B6887_2023102784235_agent]: FAILED! => { 10:52:28 "changed": true, 10:52:28 "cmd": [ 10:52:28 "/usr/local/bin/python3.10", 10:52:28 "-m", 10:52:28 "pytest", 10:52:28 "--tb=short", 10:52:28 "-v", 10:52:28 "/wazuh-qa/tests/integration/test_service.py" 10:52:28 ], 10:52:28 "delta": "0:00:00.446686", 10:52:28 "end": "2023-10-27 08:52:28.385933", 10:52:28 "invocation": { 10:52:28 "module_args": { 10:52:28 "_raw_params": "/usr/local/bin/python3.10 -m pytest --tb=short -v /wazuh-qa/tests/integration/test_service.py", 10:52:28 "_uses_shell": false, 10:52:28 "argv": null, 10:52:28 "chdir": null, 10:52:28 "creates": null, 10:52:28 "executable": null, 10:52:28 "removes": null, 10:52:28 "stdin": null, 10:52:28 "stdin_add_newline": true, 10:52:28 "strip_empty_ends": true, 10:52:28 "warn": true 10:52:28 } 10:52:28 }, 10:52:28 "rc": 4, 10:52:28 "start": "2023-10-27 08:52:27.939247" 10:52:28 } 10:52:28 10:52:28 STDERR: 10:52:28 10:52:28 ImportError while loading conftest '/wazuh-qa/tests/integration/conftest.py'. 10:52:28 /wazuh-qa/tests/integration/conftest.py:18: in 10:52:28 from wazuh_testing import ALERTS_JSON_PATH, ARCHIVES_JSON_PATH, ARCHIVES_LOG_PATH, global_parameters, logger, mocking 10:52:28 /usr/local/lib/python3.10/site-packages/wazuh_testing/mocking/__init__.py:5: in 10:52:28 from wazuh_testing.db_interface import global_db 10:52:28 /usr/local/lib/python3.10/site-packages/wazuh_testing/db_interface/__init__.py:5: in 10:52:28 import sqlite3 10:52:28 /usr/local/lib/python3.10/sqlite3/__init__.py:23: in 10:52:28 from sqlite3.dbapi2 import * 10:52:28 /usr/local/lib/python3.10/sqlite3/dbapi2.py:27: in 10:52:28 from _sqlite3 import * 10:52:28 E ModuleNotFoundError: No module named '_sqlite3' ```
pro-akim commented 6 months ago

Update

Moved to on-hold for release testing

pro-akim commented 6 months ago

Update

CentOS 7 :green_circle: build: https://ci.wazuh.info/job/Test_service/6889/

Ubuntu bionic :green_circle: build: https://ci.wazuh.info/job/Test_service/6907/console

CentOS6 and trusty :red_circle:
AMIs require adjustments.

The problem in CentOS6 and trusty openssl11 is not available and the following error appears:

``` WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. ```
pro-akim commented 6 months ago

Summary

What was done:

What is left to do:

In test_service:

To be analyzed:

In test_service:

Other pipelines/tests:

pro-akim commented 6 months ago

Upgrade

CentOS 6: Solving openssl issue by installing openssl11 separately Now new problem is appearing

``` 19:58:46 :stderr: error: subprocess-exited-with-error 19:58:46 19:58:46 × pip subprocess to install build dependencies did not run successfully. 19:58:46 │ exit code: 1 19:58:46 ╰─> [47 lines of output] 19:58:46 Collecting certifi>=2020.06.20 19:58:46 Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB) 19:58:46 Collecting numpy>=1.25 19:58:46 Downloading numpy-1.26.1.tar.gz (15.7 MB) 19:58:46 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 15.7/15.7 MB 4.7 MB/s eta 0:00:00 19:58:46 Installing build dependencies: started 19:58:46 Installing build dependencies: finished with status 'done' 19:58:46 Getting requirements to build wheel: started 19:58:46 Getting requirements to build wheel: finished with status 'done' 19:58:46 Installing backend dependencies: started 19:58:46 Installing backend dependencies: finished with status 'done' 19:58:46 Preparing metadata (pyproject.toml): started 19:58:46 Preparing metadata (pyproject.toml): finished with status 'error' 19:58:46 error: subprocess-exited-with-error 19:58:46 19:58:46 × Preparing metadata (pyproject.toml) did not run successfully. 19:58:46 │ exit code: 1 19:58:46 ╰─> [19 lines of output] 19:58:46 + /usr/local/bin/python3.10 /tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a/vendored-meson/meson/meson.py setup /tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a /tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a/.mesonpy-23dprcvl/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a/.mesonpy-23dprcvl/build/meson-python-native-file.ini 19:58:46 The Meson build system 19:58:46 Version: 1.2.99 19:58:46 Source dir: /tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a 19:58:46 Build dir: /tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a/.mesonpy-23dprcvl/build 19:58:46 Build type: native build 19:58:46 Project name: NumPy 19:58:46 Project version: 1.26.1 19:58:46 C compiler for the host machine: cc (gcc 4.4.7 "cc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23)") 19:58:46 C linker for the host machine: cc ld.bfd 2.20.51.0.2-5 19:58:46 C++ compiler for the host machine: c++ (gcc 4.4.7 "c++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23)") 19:58:46 C++ linker for the host machine: c++ ld.bfd 2.20.51.0.2-5 19:58:46 Cython compiler for the host machine: cython (cython 3.0.5) 19:58:46 Host machine cpu family: x86_64 19:58:46 Host machine cpu: x86_64 19:58:46 19:58:46 ../../meson.build:28:4: ERROR: Problem encountered: NumPy requires GCC >= 8.4 19:58:46 19:58:46 A full log can be found at /tmp/pip-install-umxxbrjt/numpy_cf3d39b656474e2191438becf838090a/.mesonpy-23dprcvl/build/meson-logs/meson-log.txt 19:58:46 [end of output] 19:58:46 19:58:46 note: This error originates from a subprocess, and is likely not a problem with pip. 19:58:46 error: metadata-generation-failed 19:58:46 19:58:46 × Encountered error while generating package metadata. 19:58:46 ╰─> See above for output. 19:58:46 19:58:46 note: This is an issue with the package mentioned above, not pip. 19:58:46 hint: See above for details. 19:58:46 [end of output] 19:58:46 19:58:46 note: This error originates from a subprocess, and is likely not a problem with pip. 19:58:46 error: subprocess-exited-with-error 19:58:46 19:58:46 × pip subprocess to install build dependencies did not run successfully. 19:58:46 │ exit code: 1 19:58:46 ╰─> See above for output. 19:58:46 19:58:46 note: This error originates from a subprocess, and is likely not a problem with pip. ```

After solving the problem of GCC installing GCC packages from scratch, now this new problem has appeared

``` ../../numpy/meson.build:169:4: ERROR: Problem encountered: No BLAS library detected! Install one, or use the `allow-noblas` build option (note, this may be up to 100x slower for some linear algebra operations). ```
pro-akim commented 6 months ago

Update

Ubuntu trusty: Python3.10 install presents the following problem

``` ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi Fatal Python error: _PySys_InitCore: can't initialize sys module Python runtime state: preinitialized Current thread 0x00002b0f9d1c8e40 (most recent call first): generate-posix-vars failed make[1]: *** [pybuilddir.txt] Error 1 make[1]: Leaving directory `/home/ubuntu/Python-3.10.0' make: *** [profile-opt] Error 2 root@ip-172-31-31-107:/home/ubuntu/Python-3.10.0# make altinstall if test "no-framework" = "no-framework" ; then \ /usr/bin/install -c python /usr/local/bin/python3.10; \ else \ /usr/bin/install -c -s Mac/pythonw /usr/local/bin/python3.10; \ fi if test "3.10" != "3.10"; then \ if test -f /usr/local/bin/python3.10 -o -h /usr/local/bin/python3.10; \ then rm -f /usr/local/bin/python3.10; \ fi; \ (cd /usr/local/bin; ln python3.10 python3.10); \ fi if test "x" != "x" ; then \ rm -f /usr/local/binpython3.10-32; \ lipo \ -output /usr/local/bin/python3.10-32 \ /usr/local/bin/python3.10; \ fi if test "x" != "x" ; then \ rm -f /usr/local/binpython3.10-intel64; \ lipo \ -output /usr/local/bin/python3.10-intel64 \ /usr/local/bin/python3.10; \ fi ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi Fatal Python error: _PySys_InitCore: can't initialize sys module Python runtime state: preinitialized Current thread 0x00002b69b2ee1e40 (most recent call first): generate-posix-vars failed make: *** [pybuilddir.txt] Error 1 root@ip-172-31-31-107:/home/ubuntu/Python-3.10.0# exit exit ```
pro-akim commented 6 months ago

Update

While I am trying to find the setting to run Python > 3.7 in Ubuntu trusty and Centos 6, the Reliability test will be research

test/pipeline status
quality>tests>generic>08-service>linux>test>conf>test-manager>tasks>main.yaml :yellow_circle:
quality>tests>generic>08-service>linux>test>conf>test-agent>tasks>main.yaml :yellow_circle:
quality>tests>reliability>agent_provision.yaml :yellow_circle:
quality>tests>reliability>manager_privision.yaml :yellow_circle:
quality>tests>reliability>test_reliability.yam :yellow_circle:
pro-akim commented 6 months ago

Update

Working on release testing. Moved to on hold

pro-akim commented 6 months ago

Update

After some additional research

In progress :yellow_circle:

file python-vers
quality>tests>generic>08-service>linux>test>conf>test-manager>tasks>main.yaml NA
quality>tests>generic>08-service>linux>test>conf>test-agent>tasks>main.yaml NA

CentOS6 and Ubuntu Trusty have problems installing new versions >=3.7 of python :red_circle:

It should be researched :yellow_circle:

file python-vers
quality>tests>performance>fim>roles>test-fim>tasks>configure_agent.yaml
quality>tests>performance>fim>roles>test-fim>tasks>configure_manager.yaml NA
quality>tests>performance>fim>roles>test-fim>tasks>configure_win_agent.yaml NA
quality>tests>performance>cluster>cluster_tests.sh NA
quality>tests>performance>cluster>manager_provision.yaml NA
jenkins-files>tests>test_e2e_system.groovy NA
jenkins-files>tests>test_integration.groovy NA

It is already solved becase they use python>=3.70 :green_circle:

file python-vers
dockers>repositories>wazuh-jenkins>aws_docker_instance>Dockerfile 3.716
doc>Dockerfile 3.7-alpine
dockers>repositories>wazuh-jenkins>data_visualization>Dockerfile 3.9
dockers>repositories>wazuh-jenkins>data_visualization>data_visualizacion.sh NA
dockers>repositories>wazuh>api_performance>Dockerfile 3.9
dockers>repositories>wazuh>api_performance>run_api_performance_test.sh NA
dockers>repositories>wazuh>simulate_agents>Dockerfile 3.9
dockers>repositories>wazuh>simulate_agents>run_simulated_agents.sh NA
dockers>repositories>wazuh>simulate_api>Dockerfile 3.9
dockers>repositories>wazuh>simulate_api>run_api_simulation.sh NA
dockers>repositoriesreport_generator>Dockerfile 3.9-buster
dockers>repositories>wazuh-jenkins>report_generator>report_generator.sh NA
dockers>repositories>wazuh-jenkins>log_parser>Dockerfile 3.9
dockers>repositories>wazuh-jenkins>log_parser>log_parser.sh NA

It depends on pipeline which is under development :white_circle:

file python-vers
quality>tests>system>playbooks>provision.yaml NA

Deprecated pipelines :white_circle:

file python-vers
quality>tests>reliability>agent_provision.yaml NA
quality>tests>reliability>manager_privision.yaml NA
quality>tests>reliability>test_reliability.yaml NA
pro-akim commented 6 months ago

Update

Test_reliability and CLUSTER-Workload_benchmarks_metrics are deprecated, then fixes on them are going to be skipped

Researching the following pipeline

file python-vers
quality>tests>performance>fim>roles>test-fim>tasks>configure_agent.yaml
quality>tests>performance>fim>roles>test-fim>tasks>configure_manager.yaml NA
quality>tests>performance>fim>roles>test-fim>tasks>configure_win_agent.yaml NA
quality>tests>performance>cluster>manager_provision.yaml NA
pro-akim commented 6 months ago

Update

It is possible to confirm the following python version in test_e2e_system and test_integration

file python-vers
jenkins-files>tests>test_e2e_system.groovy 3.9/3.11
jenkins-files>tests>test_integration.groovy 3.10.9
pro-akim commented 6 months ago

Summary

Update

After some additional research

In progress :yellow_circle:

file python-vers
quality>tests>generic>08-service>linux>test>conf>test-manager>tasks>main.yaml NA
quality>tests>generic>08-service>linux>test>conf>test-agent>tasks>main.yaml NA

CentOS6 and Ubuntu Trusty have problems installing new versions >=3.7 of python :red_circle:

It is already solved becase they use python>=3.70 :green_circle:

file python-vers
dockers>repositories>wazuh-jenkins>aws_docker_instance>Dockerfile 3.716
doc>Dockerfile 3.7-alpine
dockers>repositories>wazuh-jenkins>data_visualization>Dockerfile 3.9
dockers>repositories>wazuh-jenkins>data_visualization>data_visualizacion.sh NA
dockers>repositories>wazuh>api_performance>Dockerfile 3.9
dockers>repositories>wazuh>api_performance>run_api_performance_test.sh NA
dockers>repositories>wazuh>simulate_agents>Dockerfile 3.9
dockers>repositories>wazuh>simulate_agents>run_simulated_agents.sh NA
dockers>repositories>wazuh>simulate_api>Dockerfile 3.9
dockers>repositories>wazuh>simulate_api>run_api_simulation.sh NA
dockers>repositoriesreport_generator>Dockerfile 3.9-buster
dockers>repositories>wazuh-jenkins>report_generator>report_generator.sh NA
dockers>repositories>wazuh-jenkins>log_parser>Dockerfile 3.9
dockers>repositories>wazuh-jenkins>log_parser>log_parser.sh NA
jenkins-files>tests>test_e2e_system.groovy 3.9/3.11
jenkins-files>tests>test_integration.groovy 3.10.9

It depends on pipeline which is under development :white_circle:

file python-vers
quality>tests>system>playbooks>provision.yaml NA

Deprecated pipelines or pipes without use :white_circle:

file python-vers
quality>tests>reliability>agent_provision.yaml NA
quality>tests>reliability>manager_privision.yaml NA
quality>tests>reliability>test_reliability.yaml NA
quality>tests>performance>fim>roles>test-fim>tasks>configure_agent.yaml
quality>tests>performance>fim>roles>test-fim>tasks>configure_manager.yaml NA
quality>tests>performance>fim>roles>test-fim>tasks>configure_win_agent.yaml NA
quality>tests>performance>cluster>cluster_tests.sh NA
quality>tests>performance>cluster>manager_provision.yaml NA
pro-akim commented 6 months ago

Update

AMI for Ubuntu trusty could be created.

Centos6 is still showing WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. error

pro-akim commented 6 months ago

Update

Considering the following documents in order to work on openssl issue:

https://serverok.in/install-python-3-8-on-centos-6-from-source https://techglimpse.com/install-python-openssl-support-tutorial/

Some tests were done but some errors appeared while python is being installed changing /Modules/Setup information Openssl is being installed successfully, only using yum install openssl and yum install openssl-devel

pro-akim commented 5 months ago

Update

Working on CentOS6 it was possible to install openssl in docker, however, the following error is showed installing pip libraries

``` error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [19 lines of output] The Meson build system Version: 1.3.0 Source dir: /tmp/pip-install-f1q6lh3h/scipy_e7763ddca71e4d7d9d067f91bd534c38 Build dir: /tmp/pip-install-f1q6lh3h/scipy_e7763ddca71e4d7d9d067f91bd534c38/.mesonpy-rgcrnfp5/build Build type: native build Project name: SciPy Project version: 1.10.1 C compiler for the host machine: cc (gcc 4.4.7 "cc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23)") C linker for the host machine: cc ld.bfd 2.20.51.0.2-5 C++ compiler for the host machine: c++ (gcc 4.4.7 "c++ (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23)") C++ linker for the host machine: c++ ld.bfd 2.20.51.0.2-5 Cython compiler for the host machine: cython (cython 0.29.36) Host machine cpu family: x86_64 Host machine cpu: x86_64 ../../meson.build:26:4: ERROR: Problem encountered: SciPy requires GCC >= 8.0 A full log can be found at /tmp/pip-install-f1q6lh3h/scipy_e7763ddca71e4d7d9d067f91bd534c38/.mesonpy-rgcrnfp5/build/meson-logs/meson-log.txt + meson setup --prefix=/usr/local /tmp/pip-install-f1q6lh3h/scipy_e7763ddca71e4d7d9d067f91bd534c38 /tmp/pip-install-f1q6lh3h/scipy_e7763ddca71e4d7d9d067f91bd534c38/.mesonpy-rgcrnfp5/build --native-file=/tmp/pip-install-f1q6lh3h/scipy_e7763ddca71e4d7d9d067f91bd534c38/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. ```

Test_service AMI has GCC 7.3.1

pro-akim commented 5 months ago

Update

Ubuntu Trusty build was successful: https://ci.wazuh.info/job/Test_service/6999/console

In case of CentOS:

In docker:

After update GCC. During the python libraries installation:

``` × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [40 lines of output] The Meson build system Version: 1.3.0 Source dir: /tmp/pip-install-dk7nswbd/scipy_056ac8ebaf2f4c829e4ed5e7afa4466f Build dir: /tmp/pip-install-dk7nswbd/scipy_056ac8ebaf2f4c829e4ed5e7afa4466f/.mesonpy-42vu8bib/build Build type: native build Project name: SciPy Project version: 1.10.1 C compiler for the host machine: cc (gcc 8.1.0 "cc (GCC) 8.1.0") C linker for the host machine: cc ld.bfd 2.20.51.0.2-5 C++ compiler for the host machine: c++ (gcc 8.1.0 "c++ (GCC) 8.1.0") C++ linker for the host machine: c++ ld.bfd 2.20.51.0.2-5 Cython compiler for the host machine: cython (cython 0.29.36) Host machine cpu family: x86_64 Host machine cpu: x86_64 Compiler for C supports arguments -Wno-unused-but-set-variable: YES Compiler for C supports arguments -Wno-unused-function: YES Compiler for C supports arguments -Wno-conversion: YES Compiler for C supports arguments -Wno-misleading-indentation: YES Compiler for C supports arguments -Wno-incompatible-pointer-types: YES Library m found: YES ../../meson.build:63:0: ERROR: Unknown compiler(s): [['gfortran'], ['flang'], ['nvfortran'], ['pgfortran'], ['ifort'], ['ifx'], ['g95']] The following exception(s) were encountered: Running `gfortran --version` gave "[Errno 2] No such file or directory: 'gfortran'" Running `gfortran -V` gave "[Errno 2] No such file or directory: 'gfortran'" Running `flang --version` gave "[Errno 2] No such file or directory: 'flang'" Running `flang -V` gave "[Errno 2] No such file or directory: 'flang'" Running `nvfortran --version` gave "[Errno 2] No such file or directory: 'nvfortran'" Running `nvfortran -V` gave "[Errno 2] No such file or directory: 'nvfortran'" Running `pgfortran --version` gave "[Errno 2] No such file or directory: 'pgfortran'" Running `pgfortran -V` gave "[Errno 2] No such file or directory: 'pgfortran'" Running `ifort --version` gave "[Errno 2] No such file or directory: 'ifort'" Running `ifort -V` gave "[Errno 2] No such file or directory: 'ifort'" Running `ifx --version` gave "[Errno 2] No such file or directory: 'ifx'" Running `ifx -V` gave "[Errno 2] No such file or directory: 'ifx'" Running `g95 --version` gave "[Errno 2] No such file or directory: 'g95'" Running `g95 -V` gave "[Errno 2] No such file or directory: 'g95'" A full log can be found at /tmp/pip-install-dk7nswbd/scipy_056ac8ebaf2f4c829e4ed5e7afa4466f/.mesonpy-42vu8bib/build/meson-logs/meson-log.txt + meson setup --prefix=/usr/local /tmp/pip-install-dk7nswbd/scipy_056ac8ebaf2f4c829e4ed5e7afa4466f /tmp/pip-install-dk7nswbd/scipy_056ac8ebaf2f4c829e4ed5e7afa4466f/.mesonpy-42vu8bib/build --native-file=/tmp/pip-install-dk7nswbd/scipy_056ac8ebaf2f4c829e4ed5e7afa4466f/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. ```

Then it was installed gcc-gfortran

``` Collecting scipy>=1.0 (from -r requirements.txt (line 29)) Using cached scipy-1.10.1.tar.gz (42.4 MB) Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing metadata (pyproject.toml) ... error error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [41 lines of output] The Meson build system Version: 1.3.0 Source dir: /tmp/pip-install-0nq9z1am/scipy_060bdb2e80ff4165a6d940ecfb8c4b3e Build dir: /tmp/pip-install-0nq9z1am/scipy_060bdb2e80ff4165a6d940ecfb8c4b3e/.mesonpy-2klhezu2/build Build type: native build Project name: SciPy Project version: 1.10.1 C compiler for the host machine: cc (gcc 8.1.0 "cc (GCC) 8.1.0") C linker for the host machine: cc ld.bfd 2.20.51.0.2-5 C++ compiler for the host machine: c++ (gcc 8.1.0 "c++ (GCC) 8.1.0") C++ linker for the host machine: c++ ld.bfd 2.20.51.0.2-5 Cython compiler for the host machine: cython (cython 0.29.36) Host machine cpu family: x86_64 Host machine cpu: x86_64 Compiler for C supports arguments -Wno-unused-but-set-variable: YES Compiler for C supports arguments -Wno-unused-function: YES Compiler for C supports arguments -Wno-conversion: YES Compiler for C supports arguments -Wno-misleading-indentation: YES Compiler for C supports arguments -Wno-incompatible-pointer-types: YES Library m found: YES Fortran compiler for the host machine: gfortran (gcc 4.4.7 "GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23)") Fortran linker for the host machine: gfortran ld.bfd 2.20.51.0.2-5 Compiler for Fortran supports arguments -Wno-conversion: YES Checking if "-Wl,--version-script" : links: YES Program cython found: YES (/tmp/pip-build-env-fux5srow/overlay/bin/cython) Program python found: YES (/usr/local/bin/python3.8) Found pkg-config: YES (/usr/bin/pkg-config) 0.23 Run-time dependency python found: YES 3.8 Program pythran found: YES (/tmp/pip-build-env-fux5srow/overlay/bin/pythran) Run-time dependency threads found: YES Library npymath found: YES Library npyrandom found: YES Did not find CMake 'cmake' Found CMake: NO Run-time dependency openblas found: NO (tried pkgconfig and cmake) Run-time dependency openblas found: NO (tried pkgconfig) ../../scipy/meson.build:134:7: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig A full log can be found at /tmp/pip-install-0nq9z1am/scipy_060bdb2e80ff4165a6d940ecfb8c4b3e/.mesonpy-2klhezu2/build/meson-logs/meson-log.txt + meson setup --prefix=/usr/local /tmp/pip-install-0nq9z1am/scipy_060bdb2e80ff4165a6d940ecfb8c4b3e /tmp/pip-install-0nq9z1am/scipy_060bdb2e80ff4165a6d940ecfb8c4b3e/.mesonpy-2klhezu2/build --native-file=/tmp/pip-install-0nq9z1am/scipy_060bdb2e80ff4165a6d940ecfb8c4b3e/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. ```

Then it was installed openblas

``` error: subprocess-exited-with-error × Preparing metadata (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [42 lines of output] The Meson build system Version: 1.3.0 Source dir: /tmp/pip-install-ot6jjss6/scipy_61ecfe7b866448c69d0e09a2431a9915 Build dir: /tmp/pip-install-ot6jjss6/scipy_61ecfe7b866448c69d0e09a2431a9915/.mesonpy-rl0qg_2f/build Build type: native build Project name: SciPy Project version: 1.10.1 C compiler for the host machine: cc (gcc 8.1.0 "cc (GCC) 8.1.0") C linker for the host machine: cc ld.bfd 2.20.51.0.2-5 C++ compiler for the host machine: c++ (gcc 8.1.0 "c++ (GCC) 8.1.0") C++ linker for the host machine: c++ ld.bfd 2.20.51.0.2-5 Cython compiler for the host machine: cython (cython 0.29.36) Host machine cpu family: x86_64 Host machine cpu: x86_64 Compiler for C supports arguments -Wno-unused-but-set-variable: YES Compiler for C supports arguments -Wno-unused-function: YES Compiler for C supports arguments -Wno-conversion: YES Compiler for C supports arguments -Wno-misleading-indentation: YES Compiler for C supports arguments -Wno-incompatible-pointer-types: YES Library m found: YES Fortran compiler for the host machine: gfortran (gcc 4.4.7 "GNU Fortran (GCC) 4.4.7 20120313 (Red Hat 4.4.7-23)") Fortran linker for the host machine: gfortran ld.bfd 2.20.51.0.2-5 Compiler for Fortran supports arguments -Wno-conversion: YES Checking if "-Wl,--version-script" : links: YES Program cython found: YES (/tmp/pip-build-env-uf9miw01/overlay/bin/cython) Program python found: YES (/usr/local/bin/python3.8) Found pkg-config: YES (/usr/bin/pkg-config) 0.23 Run-time dependency python found: YES 3.8 Program pythran found: YES (/tmp/pip-build-env-uf9miw01/overlay/bin/pythran) Run-time dependency threads found: YES Library npymath found: YES Library npyrandom found: YES Found CMake: /usr/bin/cmake (2.8.12.2) WARNING: The version of CMake /usr/bin/cmake is 2.8.12.2 but version >=3.4 is required Run-time dependency openblas found: NO (tried pkgconfig and cmake) WARNING: The version of CMake /usr/bin/cmake is 2.8.12.2 but version >=3.4 is required Run-time dependency openblas found: NO (tried pkgconfig) ../../scipy/meson.build:134:7: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig A full log can be found at /tmp/pip-install-ot6jjss6/scipy_61ecfe7b866448c69d0e09a2431a9915/.mesonpy-rl0qg_2f/build/meson-logs/meson-log.txt + meson setup --prefix=/usr/local /tmp/pip-install-ot6jjss6/scipy_61ecfe7b866448c69d0e09a2431a9915 /tmp/pip-install-ot6jjss6/scipy_61ecfe7b866448c69d0e09a2431a9915/.mesonpy-rl0qg_2f/build --native-file=/tmp/pip-install-ot6jjss6/scipy_61ecfe7b866448c69d0e09a2431a9915/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 [end of output] note: This error originates from a subprocess, and is likely not a problem with pip. error: metadata-generation-failed × Encountered error while generating package metadata. ╰─> See above for output. note: This is an issue with the package mentioned above, not pip. hint: See above for details. ```

Then trying to install cmake >3.4

``` CMake 3.15.0, Copyright 2000-2019 Kitware, Inc. and Contributors Found GNU toolchain C compiler on this system is: gcc C++ compiler on this system is: g++ -std=gnu++11 Makefile processor on this system is: gmake g++ has setenv g++ has unsetenv g++ does not have environ in stdlib.h g++ has stl wstring g++ has --------------------------------------------- gmake: `cmake' is up to date. /cmake-3.15.0/Bootstrap.cmk/cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /cmake-3.15.0/Bootstrap.cmk/cmake) /cmake-3.15.0/Bootstrap.cmk/cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.18' not found (required by /cmake-3.15.0/Bootstrap.cmk/cmake) /cmake-3.15.0/Bootstrap.cmk/cmake: /usr/lib64/libstdc++.so.6: version `CXXABI_1.3.5' not found (required by /cmake-3.15.0/Bootstrap.cmk/cmake) /cmake-3.15.0/Bootstrap.cmk/cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /cmake-3.15.0/Bootstrap.cmk/cmake) /cmake-3.15.0/Bootstrap.cmk/cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /cmake-3.15.0/Bootstrap.cmk/cmake) /cmake-3.15.0/Bootstrap.cmk/cmake: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /cmake-3.15.0/Bootstrap.cmk/cmake) --------------------------------------------- Error when bootstrapping CMake: Problem while running initial CMake ```

In EC2

During GCC update

``` In file included from /home/centos/gcc-8.1.0/libstdc++-v3/include/precompiled/extc++.h:83: /home/centos/gcc-8.1.0/x86_64-pc-linux-gnu/libstdc++-v3/include/ext/enc_filebuf.h:63:1: fatal error: can’t write PCH file: No space left on device } // namespace ^ compilation terminated. make[5]: *** [x86_64-pc-linux-gnu/bits/extc++.h.gch/O2g.gch] Error 1 make[5]: Leaving directory `/home/centos/gcc-8.1.0/x86_64-pc-linux-gnu/libstdc++-v3/include' make[4]: *** [all-recursive] Error 1 make[4]: Leaving directory `/home/centos/gcc-8.1.0/x86_64-pc-linux-gnu/libstdc++-v3' make[3]: *** [all] Error 2 make[3]: Leaving directory `/home/centos/gcc-8.1.0/x86_64-pc-linux-gnu/libstdc++-v3' make[2]: *** [all-stage3-target-libstdc++-v3] Error 2 make[2]: Leaving directory `/home/centos/gcc-8.1.0' make[1]: *** [stage3-bubble] Error 2 make[1]: Leaving directory `/home/centos/gcc-8.1.0' make: *** [all] Error 2 [root@ip-172-31-94-99 gcc-8.1.0]# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 7,8G 7,8G 0 100% / tmpfs 16G 0 16G 0% /dev/shm [root@ip-172-31-94-99 gcc-8.1.0]# ```

Then new EC2 with 150gb has created image

However, same error has raised by the EC2

pro-akim commented 5 months ago

Update

Same problems installing cmake. Research about the error demonstrates that the version of GLIBCXX CXXABI is present in libstdc++.so.6.

pro-akim commented 5 months ago

Update

When examining Centos 6 AMIs, it was observed that the current AMI already includes Python 3.8. When attempting to install the requirements.txt with the fixed 'setuptools' library, an error surfaced as follows:

../../meson.build:26:4: ERROR: Encountered an issue: SciPy necessitates GCC >= 8.0

Trying to install GCC >= 8.0 resulted in the same space shortage issue: link to the issue

pro-akim commented 5 months ago

Update

It is possible to see that AMIs have a specific defined volume for the EC2 root user. It should be changed manually in case that the idea is still install GCC>8.0

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html

pro-akim commented 5 months ago

Update

The same issue persists when trying to install GCC on the instances launched from AMI.

pro-akim commented 4 months ago

Update

I don't have access to CentOS 6 AMI, the task will resume when possible.

akim@akim-PC:~/Desktop/personal$ ssh -i "xx.pem" centos@ec2-xx.compute-1.amazonaws.com
}sign_and_send_pubkey: no mutual signature supported
centos@ec2-xx.compute-1.amazonaws.com: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
pro-akim commented 3 months ago

Update

It will not be worked on until there is a migration of the QA framework.

davidjiglesias commented 3 months ago

Based on the comment above:

It will not be worked on until there is a migration of the QA framework.

We will decide what to do with the repository dependencies once the migration is completed, but the solution might be different to the proposed here.