search

wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
10.46k stars 1.61k forks source link

Enhance Wazuh embedded Python interpreter installation #17454

Open Selutario opened 1 year ago

Selutario commented 1 year ago

Description

Wazuh currently stores the Python interpreter and its dependencies in ascending numbered folders. Every time that upgrading a specific library is needed, a new folder must be created, copying the contents of the previous one and replacing the library.

These folders are publicly accessible so that, during the installation or upgrade of managers, everything inside the folder whose number corresponds to the version to be installed is downloaded: https://github.com/wazuh/wazuh/blob/b197e669827fc72de31b566636e7b9b74585765f/src/Makefile#L1032 https://github.com/wazuh/wazuh/blob/5bae1c1830dbf11acc8a06e01f7a5a134b767760/src/Makefile#L1092 https://github.com/wazuh/wazuh/blob/2477e9fa50bc1424e834ac8401ce2450a5978e75/src/Makefile#L1191

This makes maintenance difficult and takes up storage space with duplicate files. To solve this, it is requested to change the mentioned structure. The goal is that each version of each dependency has its folder and, during the installation, the necessary ones are downloaded according to the requirements.txt file.

Requirements

Functional requirements

Non-functional requirements

Implementation restrictions

Plan

Selutario commented 1 year ago

Duplicated:

Selutario commented 1 year ago

I'm reopening the issue since we should still take care of the Python interpreter version and its libraries.

nico-stefani commented 4 days ago

Update

I've been investigating a way to build Python and was analyzing the possibility of using Pyenv to do it.

This tool is capable of building different versions of Python in simple steps.

POC

In a freshly installed container

  1. Install the dependencies to build Python 
    apt install build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl git \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
  2. Install Pyenv 
    curl https://pyenv.run | bash
  3. Install the desired Python version, in this case 3.10.15 
    root@2cf5bc644c8d:~# PYENV_ROOT=/root ./.pyenv/bin/pyenv install 3.10.15
Downloading Python-3.10.15.tar.xz...
-> https://www.python.org/ftp/python/3.10.15/Python-3.10.15.tar.xz
Installing Python-3.10.15...
Installed Python-3.10.15 to /root/versions/3.10.15
  4. Get the interpreter up and running 
    root@2cf5bc644c8d:~# ll versions/3.10.15/
total 24
drwxr-xr-x 6 root root 4096 Sep 18 18:16 ./
drwxr-xr-x 3 root root 4096 Sep 18 17:37 ../
drwxr-xr-x 2 root root 4096 Sep 18 17:38 bin/
drwxr-xr-x 3 root root 4096 Sep 18 17:37 include/
drwxr-xr-x 4 root root 4096 Sep 18 17:37 lib/
drwxr-xr-x 3 root root 4096 Sep 18 17:37 share/
    root@2cf5bc644c8d:~# versions/3.10.15/bin/python
Python 3.10.15 (main, Sep 18 2024, 17:37:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
  5. Move the compiled python to another machine 
    ➜  workdir cpython-from-pyenv/bin/python
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

With these steps, we compiled Python. Installing the requirements here we will obtain the pre-installed ready to distribute.

This could be a good option to include in a Github action to build Python in a few steps without the necessity to maintain a script.

nico-stefani commented 4 days ago

Trying the portability of the Pyenv generated pre-installed Python

Having installed pyenv in a CentOS container we proceed to create a pre-installated Python and test it in different OS's.

[!NOTE] Was needed to install openssl 1.1.1 manually from sources

[root@7d26c63b09f0 ~]# CONFIGURE_OPTS="--with-openssl=/usr/local/openssl --enable-shared" PYENV_ROOT=/root ./.pyenv/bin/pyenv install 3.10.15
Downloading Python-3.10.15.tar.xz...
-> https://www.python.org/ftp/python/3.10.15/Python-3.10.15.tar.xz
Installing Python-3.10.15...
Installed Python-3.10.15 to /root/versions/3.10.15
[root@7d26c63b09f0 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@7d26c63b09f0 ~]# versions/3.10.15/bin/python
Python 3.10.15 (main, Sep 19 2024, 19:36:22) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

Testing on Ubuntu

root@5cb783605e36:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@5cb783605e36:~# cpython/bin/python
cpython/bin/python: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

At first glance is not possible the get the interpreter working on a different OS with the default configurations. I'm considering here if pyenv is the right choice to use of if it just we need to do the compilation by ourself taking some of the configurations used in the current makefile.

nico-stefani commented 3 days ago

Analysis of dependencies handling

I've been analyzing the current status of the dependecies handling and I came to the conclusion that It would be much simpler if we delegate the versions resolution directly to pip through pypi.

The process, that we do today, to download de whl within the Dependencies directory It is needed because we distribute the pre-compiled for two different architectures (AMD64, AARHC64). Having in account that two of these restrictions disappear and we'll only distribute the pre-installed for AMD64 all this procees would keep obsolete.

The result is a proccess than can be put in a pipeline to the generate a pre-installed Python ready to use. Which consists of the following steps:

  1. Get the pre-compiled Python. This always will be the same unless we perform a Python version upgrade, but it could be the output of another pipeline, here we are uncoupling the two processes.
  2. Install the dependecies defined in the requirements.txt. Surely in this step we will have to deal with conflicts between dependencies but this will never be transmitted to the user.
  3. If the previous step runs OK, here we generate the pre-installed ready to use.
nico-stefani commented 3 days ago

Update

I've been trying to build Python, using as a base the current makefile command but without any of the links that we currently have to Wazuh libs.

[!NOTE] For this build I've used the OpenSSL version that is currently distributed with Wazuh

After configuring, building, and installing

export WPYTHON_DIR=/var/wazuh/framework/python
export OPENSSL_DIR=/wazuh/src/external/openssl

./configure --prefix="${WPYTHON_DIR}" --libdir="${WPYTHON_DIR}/lib" --enable-shared --with-openssl="${OPENSSL_DIR}" LDFLAGS="-Wl,-rpath,'/usr/local/lib',--disable-new-dtags" CPPFLAGS="-I${OPENSSL_DIR} && make && make install

I'm having an error with the shared modules

[root@4023100c78a1 Python-3.10.15]# /var/wazuh/framework/python/bin/python3
/var/wazuh/framework/python/bin/python3: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

Further investigation is needed to understand the meaning of the different configuration options.

fdalmaup commented 10 hours ago

Update

The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc pwd time

Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer


Which leads to an error when trying to import `ssl` (used in the cluster):

Python 3.10.15 (v3.10.15:ffee63f, Sep 23 2024, 16:00:04) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import ssl Traceback (most recent call last): File "", line 1, in File "/var/wazuh/framework/python/lib/python3.10/ssl.py", line 99, in import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl'

It is probable that some changes are needed in the python/cpython repository files before building it to obtain a running interpreter.