search

wazuh / wazuh

Wazuh - The Open Source Security Platform. Unified XDR and SIEM protection for endpoints and cloud workloads.
https://wazuh.com/
Other
11.13k stars 1.69k forks source link

Enhance Wazuh embedded Python interpreter installation #17454

Open Selutario opened 1 year ago

Selutario commented 1 year ago

Description

Wazuh currently stores the Python interpreter and its dependencies in ascending numbered folders. Every time that upgrading a specific library is needed, a new folder must be created, copying the contents of the previous one and replacing the library.

These folders are publicly accessible so that, during the installation or upgrade of managers, everything inside the folder whose number corresponds to the version to be installed is downloaded: https://github.com/wazuh/wazuh/blob/b197e669827fc72de31b566636e7b9b74585765f/src/Makefile#L1032 https://github.com/wazuh/wazuh/blob/5bae1c1830dbf11acc8a06e01f7a5a134b767760/src/Makefile#L1092 https://github.com/wazuh/wazuh/blob/2477e9fa50bc1424e834ac8401ce2450a5978e75/src/Makefile#L1191

This makes maintenance difficult and takes up storage space with duplicate files. To solve this, it is requested to change the mentioned structure. The goal is that each version of each dependency has its folder and, during the installation, the necessary ones are downloaded according to the requirements.txt file.

Requirements

Functional requirements

Non-functional requirements

Implementation restrictions

Plan

Selutario commented 1 year ago

Duplicated:

Selutario commented 1 year ago

I'm reopening the issue since we should still take care of the Python interpreter version and its libraries.

nico-stefani commented 2 months ago

Update

I've been investigating a way to build Python and was analyzing the possibility of using Pyenv to do it.

This tool is capable of building different versions of Python in simple steps.

POC

In a freshly installed container

  1. Install the dependencies to build Python 
    apt install build-essential libssl-dev zlib1g-dev \
libbz2-dev libreadline-dev libsqlite3-dev curl git \
libncursesw5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev
  2. Install Pyenv 
    curl https://pyenv.run | bash
  3. Install the desired Python version, in this case 3.10.15 
    root@2cf5bc644c8d:~# PYENV_ROOT=/root ./.pyenv/bin/pyenv install 3.10.15
Downloading Python-3.10.15.tar.xz...
-> https://www.python.org/ftp/python/3.10.15/Python-3.10.15.tar.xz
Installing Python-3.10.15...
Installed Python-3.10.15 to /root/versions/3.10.15
  4. Get the interpreter up and running 
    root@2cf5bc644c8d:~# ll versions/3.10.15/
total 24
drwxr-xr-x 6 root root 4096 Sep 18 18:16 ./
drwxr-xr-x 3 root root 4096 Sep 18 17:37 ../
drwxr-xr-x 2 root root 4096 Sep 18 17:38 bin/
drwxr-xr-x 3 root root 4096 Sep 18 17:37 include/
drwxr-xr-x 4 root root 4096 Sep 18 17:37 lib/
drwxr-xr-x 3 root root 4096 Sep 18 17:37 share/
    root@2cf5bc644c8d:~# versions/3.10.15/bin/python
Python 3.10.15 (main, Sep 18 2024, 17:37:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
  5. Move the compiled python to another machine 
    ➜  workdir cpython-from-pyenv/bin/python
Python 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

With these steps, we compiled Python. Installing the requirements here we will obtain the pre-installed ready to distribute.

This could be a good option to include in a Github action to build Python in a few steps without the necessity to maintain a script.

nico-stefani commented 2 months ago

Trying the portability of the Pyenv generated pre-installed Python

Having installed pyenv in a CentOS container we proceed to create a pre-installated Python and test it in different OS's.

[!NOTE] Was needed to install openssl 1.1.1 manually from sources

[root@7d26c63b09f0 ~]# CONFIGURE_OPTS="--with-openssl=/usr/local/openssl --enable-shared" PYENV_ROOT=/root ./.pyenv/bin/pyenv install 3.10.15
Downloading Python-3.10.15.tar.xz...
-> https://www.python.org/ftp/python/3.10.15/Python-3.10.15.tar.xz
Installing Python-3.10.15...
Installed Python-3.10.15 to /root/versions/3.10.15
[root@7d26c63b09f0 ~]# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

[root@7d26c63b09f0 ~]# versions/3.10.15/bin/python
Python 3.10.15 (main, Sep 19 2024, 19:36:22) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

Testing on Ubuntu

root@5cb783605e36:~# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@5cb783605e36:~# cpython/bin/python
cpython/bin/python: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

At first glance is not possible the get the interpreter working on a different OS with the default configurations. I'm considering here if pyenv is the right choice to use of if it just we need to do the compilation by ourself taking some of the configurations used in the current makefile.

nico-stefani commented 2 months ago

Analysis of dependencies handling

I've been analyzing the current status of the dependecies handling and I came to the conclusion that It would be much simpler if we delegate the versions resolution directly to pip through pypi.

The process, that we do today, to download de whl within the Dependencies directory It is needed because we distribute the pre-compiled for two different architectures (AMD64, AARHC64). Having in account that two of these restrictions disappear and we'll only distribute the pre-installed for AMD64 all this procees would keep obsolete.

The result is a proccess than can be put in a pipeline to the generate a pre-installed Python ready to use. Which consists of the following steps:

  1. Get the pre-compiled Python. This always will be the same unless we perform a Python version upgrade, but it could be the output of another pipeline, here we are uncoupling the two processes.
  2. Install the dependecies defined in the requirements.txt. Surely in this step we will have to deal with conflicts between dependencies but this will never be transmitted to the user.
  3. If the previous step runs OK, here we generate the pre-installed ready to use.
nico-stefani commented 2 months ago

Update

I've been trying to build Python, using as a base the current makefile command but without any of the links that we currently have to Wazuh libs.

[!NOTE] For this build I've used the OpenSSL version that is currently distributed with Wazuh

After configuring, building, and installing

export WPYTHON_DIR=/var/wazuh/framework/python
export OPENSSL_DIR=/wazuh/src/external/openssl

./configure --prefix="${WPYTHON_DIR}" --libdir="${WPYTHON_DIR}/lib" --enable-shared --with-openssl="${OPENSSL_DIR}" LDFLAGS="-Wl,-rpath,'/usr/local/lib',--disable-new-dtags" CPPFLAGS="-I${OPENSSL_DIR} && make && make install

I'm having an error with the shared modules

[root@4023100c78a1 Python-3.10.15]# /var/wazuh/framework/python/bin/python3
/var/wazuh/framework/python/bin/python3: error while loading shared libraries: libpython3.10.so.1.0: cannot open shared object file: No such file or directory

Further investigation is needed to understand the meaning of the different configuration options.

fdalmaup commented 2 months ago

Update

The following modules found by detect_modules() in setup.py, have been built by the Makefile instead, as configured by the Setup files: _abc pwd time

Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer


Which leads to an error when trying to import `ssl` (used in the cluster):

Python 3.10.15 (v3.10.15:ffee63f, Sep 23 2024, 16:00:04) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information.

import ssl Traceback (most recent call last): File "", line 1, in File "/var/wazuh/framework/python/lib/python3.10/ssl.py", line 99, in import _ssl # if we can't import it, let the error propagate ModuleNotFoundError: No module named '_ssl'

It is probable that some changes are needed in the python/cpython repository files before building it to obtain a running interpreter.

fdalmaup commented 2 months ago

Update

fdalmaup commented 2 months ago

Update

Pyinstaller to generate an executable of wazuh-server

I have been reading and testing Pyinstaller, a Python package that bundles a Python application and all its dependencies into a single package for various operating systems. It takes a Python script and generates a single executable file that contains all the necessary dependencies and can be run on computers that do not have Python installed. The hypothesis was to generate the wazuh-server executable that could be installed in any of the Wazuh 5.0 supported OSs. It has the limitation that it does not bundle everything the application needs to run. The executable is still dependent on the users’ glibc. This can be worked around by building on the oldest version of each OS intended to target. If we want to target a wide array of Linux machines, then we should build on CentOS 7, which is the approach currently used for the compilation of the Python interpreter. Therefore we will still be facing the same challenges related to the Python 3.10 build in CentOS 7.

Installation of Python 3.10.15 in CentOS 7

Using the provided compressed source code

wget https://www.python.org/ftp/python/3.10.15/Python-3.10.15.tgz
# After decompression

./configure --enable-optimizations
make altinstall

An error arises when trying to import the ssl module:

[root@61df9b056d1e Python-3.10.15]# python3.10 
Python 3.10.15 (main, Sep 26 2024, 09:12:11) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/ssl.py", line 99, in <module>
    import _ssl             # if we can't import it, let the error propagate
ModuleNotFoundError: No module named '_ssl'
>>> exit()

This affects installing dependencies:

# pip3.10 install requests
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/requests/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/requests/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/requests/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/requests/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.")': /simple/requests/
Could not fetch URL https://pypi.org/simple/requests/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/requests/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping
ERROR: Could not find a version that satisfies the requirement requests (from versions: none)
ERROR: No matching distribution found for requests
WARNING: pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.
Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.")) - skipping

Libraries required for the Wazuh Python interpreter in CentOS 7

fdalmaup commented 2 months ago

Update

I have been trying the Pyenv approach carried out by @nico-stefani at https://github.com/wazuh/wazuh/issues/17454#issuecomment-2362086145. The error he mentioned arises because the dynamic linker (responsible for loading shared libraries) cannot find the libpython3.10.so.1.0 library. This can be fixed by adding the directory where the library is located to the LD_LIBRARY_PATH environment variable:

export LD_LIBRARY_PATH=/cpython/lib:$LD_LIBRARY_PATH

Once executed, the error is related to the ssl module:

[root@54b418f96891 cpython]# export LD_LIBRARY_PATH=/cpython/lib:$LD_LIBRARY_PATH
[root@54b418f96891 cpython]# ./bin/python3
Python 3.10.15 (main, Sep 27 2024, 13:26:08) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/cpython/lib/python3.10/ssl.py", line 99, in <module>
    import _ssl             # if we can't import it, let the error propagate
ImportError: libssl.so.3: cannot open shared object file: No such file or directory

This is again fixed by adding where the libssl.so.3 file is located:

 export LD_LIBRARY_PATH=/usr/lib/openssl-3.1.3/:$LD_LIBRARY_PATH
[root@54b418f96891 cpython]# ./bin/python3
Python 3.10.15 (main, Sep 27 2024, 13:26:08) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>>

The Pyenv approach can be further investigated to avoid building CPython from sources.

fdalmaup commented 2 months ago

Update

The container did not require to have installed said library.

It was also noticed that the PyYAML dependency cannot be installed directly via pip due to the following error:

Collecting PyYAML==5.4.1
  Downloading PyYAML-5.4.1.tar.gz (175 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 77.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [48 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/var/wazuh/framework/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/var/wazuh/framework/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/var/wazuh/framework/python/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 183, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 950, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 311, in run
          self.find_sources()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 319, in find_sources
          mm.run()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 540, in run
          self.add_defaults()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 108, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 238, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 323, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-m6jbf7xn/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

This has already been seen in https://github.com/wazuh/wazuh/issues/19920 because the package needs to be built.

nico-stefani commented 2 months ago

Update

I've continued the job of @fdalmaup but I got stuck trying to compile python with the next error.

rm -f libpython3.10.a
ar rcs libpython3.10.a Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/posixmodule.o  Modules/errnomodule.o  Modules/pwdmodule.o  Modules/_sre.o  Modules/_codecsmodule.o  Modules/_weakref.o  Modules/_functoolsmodule.o  Modules/_operator.o  Modules/_collectionsmodule.o  Modules/_abc.o  Modules/itertoolsmodule.o  Modules/atexitmodule.o  Modules/signalmodule.o  Modules/_stat.o  Modules/timemodule.o  Modules/_threadmodule.o  Modules/_localemodule.o  Modules/_iomodule.o Modules/iobase.o Modules/fileio.o Modules/bytesio.o Modules/bufferedio.o Modules/textio.o Modules/stringio.o  Modules/faulthandler.o  Modules/_tracemalloc.o  Modules/symtablemodule.o  Modules/_contextvarsmodule.o  Modules/_elementtree.o  Modules/_pickle.o  Modules/_datetimemodule.o  Modules/_asynciomodule.o  Modules/_json.o  Modules/fcntlmodule.o  Modules/grpmodule.o  Modules/selectmodule.o  Modules/socketmodule.o  Modules/_ssl.o  Modules/_hashopenssl.o  Modules/_posixsubprocess.o  Modules/md5module.o  Modules/sha1module.o  Modules/sha256module.o  Modules/sha512module.o  Modules/sha3module.o  Modules/blake2module.o Modules/blake2b_impl.o Modules/blake2s_impl.o  Modules/xxsubtype.o  Python/frozen.o
gcc -pthread   -fno-semantic-interposition -fprofile-generate -Xlinker -export-dynamic -o python Programs/python.o libpython3.10.a -lcrypt -lpthread -ldl  -lutil -lm -L/usr/lib/openssl/lib64 -l:libssl.a -Wl,--exclude-libs,libssl.a -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a  -L/usr/lib/openssl/lib64 -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a           -lm
gcc -pthread   -fno-semantic-interposition -fprofile-generate -Xlinker -export-dynamic -o Programs/_testembed Programs/_testembed.o libpython3.10.a -lcrypt -lpthread -ldl  -lutil -lm -L/usr/lib/openssl/lib64 -l:libssl.a -Wl,--exclude-libs,libssl.a -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a  -L/usr/lib/openssl/lib64 -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a           -lm
libpython3.10.a(config.o):(.data+0x298): undefined reference to `PyInit__sqlite3'
libpython3.10.a(config.o):(.data+0x2a8): undefined reference to `PyInit__tkinter'
libpython3.10.a(config.o):(.data+0x2b8): undefined reference to `PyInit(char, long double __restrict, short,...)(short)'
libpython3.10.a(config.o):(.data+0x2c8): undefined reference to `PyInit_pyexpat'
collect2: error: ld returned 1 exit status
make[3]: *** [Programs/_testembed] Error 1
make[3]: *** Waiting for unfinished jobs....
libpython3.10.a(config.o):(.data+0x298): undefined reference to `PyInit__sqlite3'
libpython3.10.a(config.o):(.data+0x2a8): undefined reference to `PyInit__tkinter'
libpython3.10.a(config.o):(.data+0x2b8): undefined reference to `PyInit(char, long double __restrict, short,...)(short)'
libpython3.10.a(config.o):(.data+0x2c8): undefined reference to `PyInit_pyexpat'
collect2: error: ld returned 1 exit status
make[3]: *** [python] Error 1
make[3]: Leaving directory `/root/cpython'
make[2]: *** [build_all_generate_profile] Error 2
make[2]: Leaving directory `/root/cpython'
make[1]: *** [profile-gen-stamp] Error 2
make[1]: Leaving directory `/root/cpython'
make: *** [profile-run-stamp] Error 2

I'll continue trying to work on the static linking of SQLite.

nico-stefani commented 2 months ago

Update

After installing the last version of sqlite3 

cd /tmp
wget https://www.sqlite.org/2024/sqlite-autoconf-3460100.tar.gz

tar xvf sqlite-autoconf-3460100.tar.gz

cd sqlite-autoconf*/

./configure --prefix=/usr/lib/sqlite3

make -j $(nproc)
make install

ldconfig

export PATH=/usr/lib/openssl/bin:/usr/lib/sqlite3/bin:\$PATH
export LD_LIBRARY_PATH=/usr/lib/openssl/lib:/usr/lib/openssl/lib64:/usr/lib/sqlite3/lib:\$LD_LIBRARY_PATH
root@aebb8fa4ca8c cpython]# sqlite3
SQLite version 3.46.1 2024-08-13 09:16:08
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite>

I've been able to build and install python and get it up and running into another host

[root@aebb8fa4ca8c cpython]# /var/wazuh/framework/python/bin/python3.10
Python 3.10.15 (v3.10.15-dirty:ffee63f, Oct  1 2024, 21:00:19) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
➜  python bin/python3.10
Python 3.10.15 (v3.10.15-dirty:ffee63f, Oct  1 2024, 21:00:19) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> import ssl
nico-stefani commented 2 months ago

Update

I've been cleaning the depencies keeping only the direct ones.

Direct dependecies

```diff diff --git a/framework/requirements.txt b/framework/requirements.txt index a64b0fae4f..2ef9231a71 100644 --- a/framework/requirements.txt +++ b/framework/requirements.txt @@ -1,78 +1,30 @@ -annotated-types==0.7.0 -anyio==4.1.0 aiohttp==3.9.5 -aiosignal==1.3.1 -asgiref==3.7.2 -async-timeout==4.0.3 -attrs==23.1.0 -brotli==1.1.0 brotli-asgi==1.4.0 cachetools==4.1.0 -certifi==2023.7.22 -cffi==1.15.1 chardet==3.0.4 -charset-normalizer==2.0.4 -click==8.1.3 connexion==3.1.0 content-size-limit-asgi==0.1.5 cryptography==42.0.4 Cython==0.29.36 defusedxml==0.6.0 -dnspython==2.6.1 -email_validator==2.2.0 -Events==0.5 -exceptiongroup==1.2.0 fastapi==0.111.1 -frozenlist==1.2.0 future==0.18.3 -greenlet==2.0.2 gunicorn==22.0.0 -httpcore==1.0.2 httpx==0.26.0 -h11==0.14.0 -idna==3.7.0 -inflection==0.3.1 -Jinja2==3.1.4 jsonschema==4.20.0 -jsonschema-path==0.3.2 -jsonschema-specifications==2023.11.2 -lazy-object-proxy==1.10.0 -MarkupSafe==2.1.2 more-itertools==8.2.0 -multidict==5.2.0 mypy-extensions==0.4.3 -openapi-schema-validator==0.6.2 openapi-spec-validator==0.7.1 opensearch-py==2.6.0 -packaging==20.9 -pathable==0.4.3 -pathlib==1.0.1 -pydantic==2.8.2 -pydantic_core==2.20.1 psutil==5.9.0 -pyasn1==0.4.8 -pycparser==2.21 PyJWT==2.8.0 -pyparsing==2.4.7 python-dateutil==2.8.1 python-json-logger==2.0.2 -python-multipart==0.0.9 pytz==2020.1 -PyYAML==5.4.1 -referencing==0.31.1 -requests==2.32.2 -rfc3339-validator==0.1.4 -rpds-py==0.15.2 rsa==4.7.2 secure==0.3.0 -six==1.16.0 -sniffio==1.3.0 SQLAlchemy==2.0.23 -starlette==0.37.2 -urllib3==2.2.2 uuid6==2024.07.10 uvloop==0.17.0 uvicorn==0.24.0.post1 -Werkzeug==3.0.3 xmltodict==0.12.0 -yarl==1.7.0 ```

After doing that I've been able to install it without conflicts in the pre-installed python

Installation Output

```console [root@0d6021d32bc7 pkg]# /var/wazuh/framework/python/bin/pip3.10 install -r /pkg/requirements.tx Collecting aiohttp==3.9.5 Downloading aiohttp-3.9.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.2 MB/s eta 0:00:00 Collecting brotli-asgi==1.4.0 Downloading brotli_asgi-1.4.0-py3-none-any.whl (4.6 kB) Collecting cachetools==4.1.0 Downloading cachetools-4.1.0-py3-none-any.whl (10 kB) Collecting chardet==3.0.4 Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.4/133.4 kB 1.1 MB/s eta 0:00:00 Collecting connexion==3.1.0 Downloading connexion-3.1.0-py3-none-any.whl (113 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 113.1/113.1 kB 1.2 MB/s eta 0:00:00 Collecting content-size-limit-asgi==0.1.5 Downloading content_size_limit_asgi-0.1.5-py3-none-any.whl (4.8 kB) Collecting cryptography==42.0.4 Downloading cryptography-42.0.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 1.2 MB/s eta 0:00:00 Collecting Cython==0.29.36 Downloading Cython-0.29.36-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.9/1.9 MB 1.2 MB/s eta 0:00:00 Collecting defusedxml==0.6.0 Downloading defusedxml-0.6.0-py2.py3-none-any.whl (23 kB) Collecting fastapi==0.111.1 Downloading fastapi-0.111.1-py3-none-any.whl (92 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 92.2/92.2 kB 1.1 MB/s eta 0:00:00 Collecting future==0.18.3 Downloading future-0.18.3.tar.gz (840 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 840.9/840.9 kB 1.2 MB/s eta 0:00:00 Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'done' Collecting gunicorn==22.0.0 Downloading gunicorn-22.0.0-py3-none-any.whl (84 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.4/84.4 kB 1.1 MB/s eta 0:00:00 Collecting httpx==0.26.0 Downloading httpx-0.26.0-py3-none-any.whl (75 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.9/75.9 kB 1.1 MB/s eta 0:00:00 Collecting jsonschema==4.20.0 Downloading jsonschema-4.20.0-py3-none-any.whl (84 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.7/84.7 kB 1.1 MB/s eta 0:00:00 Collecting more-itertools==8.2.0 Downloading more_itertools-8.2.0-py3-none-any.whl (43 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.2/43.2 kB 1.3 MB/s eta 0:00:00 Collecting mypy-extensions==0.4.3 Downloading mypy_extensions-0.4.3-py2.py3-none-any.whl (4.5 kB) Collecting openapi-spec-validator==0.7.1 Downloading openapi_spec_validator-0.7.1-py3-none-any.whl (38 kB) Collecting opensearch-py==2.6.0 Downloading opensearch_py-2.6.0-py2.py3-none-any.whl (311 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.2/311.2 kB 1.2 MB/s eta 0:00:00 Collecting psutil==5.9.0 Downloading psutil-5.9.0-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 281.4/281.4 kB 1.2 MB/s eta 0:00:00 Collecting PyJWT==2.8.0 Downloading PyJWT-2.8.0-py3-none-any.whl (22 kB) Collecting python-dateutil==2.8.1 Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.2/227.2 kB 1.2 MB/s eta 0:00:00 Collecting python-json-logger==2.0.2 Downloading python_json_logger-2.0.2-py3-none-any.whl (7.4 kB) Collecting pytz==2020.1 Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 510.2/510.2 kB 1.2 MB/s eta 0:00:00 Collecting rsa==4.7.2 Downloading rsa-4.7.2-py3-none-any.whl (34 kB) Collecting secure==0.3.0 Downloading secure-0.3.0-py3-none-any.whl (9.6 kB) Collecting SQLAlchemy==2.0.23 Downloading SQLAlchemy-2.0.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 1.2 MB/s eta 0:00:00 Collecting uuid6==2024.07.10 Downloading uuid6-2024.7.10-py3-none-any.whl (6.4 kB) Collecting uvloop==0.17.0 Downloading uvloop-0.17.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.1/4.1 MB 1.2 MB/s eta 0:00:00 Collecting uvicorn==0.24.0.post1 Downloading uvicorn-0.24.0.post1-py3-none-any.whl (59 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.7/59.7 kB 994.1 kB/s eta 0:00:00 Collecting xmltodict==0.12.0 Downloading xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB) Collecting frozenlist>=1.1.1 Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 239.5/239.5 kB 1.2 MB/s eta 0:00:00 Collecting async-timeout<5.0,>=4.0 Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB) Collecting multidict<7.0,>=4.5 Downloading multidict-6.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 124.6/124.6 kB 1.1 MB/s eta 0:00:00 Collecting yarl<2.0,>=1.0 Downloading yarl-1.13.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (447 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 447.9/447.9 kB 1.2 MB/s eta 0:00:00 Collecting aiosignal>=1.1.2 Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB) Collecting attrs>=17.3.0 Downloading attrs-24.2.0-py3-none-any.whl (63 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.0/63.0 kB 1.1 MB/s eta 0:00:00 Collecting starlette>=0.25.0 Downloading starlette-0.39.2-py3-none-any.whl (73 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 73.2/73.2 kB 1.1 MB/s eta 0:00:00 Collecting brotli>=1.0.9 Downloading Brotli-1.1.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.0 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 1.2 MB/s eta 0:00:00 Collecting werkzeug>=2.2.1 Downloading werkzeug-3.0.4-py3-none-any.whl (227 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 1.2 MB/s eta 0:00:00 Collecting asgiref>=3.4 Downloading asgiref-3.8.1-py3-none-any.whl (23 kB) Collecting Jinja2>=3.0.0 Downloading jinja2-3.1.4-py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.3/133.3 kB 1.2 MB/s eta 0:00:00 Collecting python-multipart>=0.0.5 Downloading python_multipart-0.0.12-py3-none-any.whl (23 kB) Collecting typing-extensions>=4.6.1 Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) Collecting PyYAML>=5.1 Downloading PyYAML-6.0.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (751 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 751.2/751.2 kB 1.2 MB/s eta 0:00:00 Collecting inflection>=0.3.1 Downloading inflection-0.5.1-py2.py3-none-any.whl (9.5 kB) Collecting requests>=2.27 Downloading requests-2.32.3-py3-none-any.whl (64 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 1.0 MB/s eta 0:00:00 Collecting cffi>=1.12 Downloading cffi-1.17.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (446 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 446.2/446.2 kB 1.2 MB/s eta 0:00:00 Collecting fastapi-cli>=0.0.2 Downloading fastapi_cli-0.0.5-py3-none-any.whl (9.5 kB) Collecting starlette>=0.25.0 Downloading starlette-0.37.2-py3-none-any.whl (71 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.9/71.9 kB 1.0 MB/s eta 0:00:00 Collecting email_validator>=2.0.0 Downloading email_validator-2.2.0-py3-none-any.whl (33 kB) Collecting uvicorn[standard]>=0.12.0 Downloading uvicorn-0.31.0-py3-none-any.whl (63 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.7/63.7 kB 1.1 MB/s eta 0:00:00 Collecting pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,!=2.1.0,<3.0.0,>=1.7.4 Downloading pydantic-2.9.2-py3-none-any.whl (434 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 434.9/434.9 kB 1.2 MB/s eta 0:00:00 Collecting packaging Downloading packaging-24.1-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 kB 1.0 MB/s eta 0:00:00 Collecting idna Downloading idna-3.10-py3-none-any.whl (70 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 70.4/70.4 kB 1.1 MB/s eta 0:00:00 Collecting httpcore==1.* Downloading httpcore-1.0.6-py3-none-any.whl (78 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.0/78.0 kB 1.3 MB/s eta 0:00:00 Collecting certifi Downloading certifi-2024.8.30-py3-none-any.whl (167 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.3/167.3 kB 1.2 MB/s eta 0:00:00 Collecting anyio Downloading anyio-4.6.0-py3-none-any.whl (89 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.6/89.6 kB 1.1 MB/s eta 0:00:00 Collecting sniffio Downloading sniffio-1.3.1-py3-none-any.whl (10 kB) Collecting referencing>=0.28.4 Downloading referencing-0.35.1-py3-none-any.whl (26 kB) Collecting jsonschema-specifications>=2023.03.6 Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB) Collecting rpds-py>=0.7.1 Downloading rpds_py-0.20.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (354 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 354.8/354.8 kB 1.2 MB/s eta 0:00:00 Collecting openapi-schema-validator<0.7.0,>=0.6.0 Downloading openapi_schema_validator-0.6.2-py3-none-any.whl (8.8 kB) Collecting lazy-object-proxy<2.0.0,>=1.7.1 Downloading lazy_object_proxy-1.10.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (68 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.3/68.3 kB 1.2 MB/s eta 0:00:00 Collecting jsonschema-path<0.4.0,>=0.3.1 Downloading jsonschema_path-0.3.3-py3-none-any.whl (14 kB) Collecting urllib3!=2.2.0,<3,>=1.26.18 Downloading urllib3-2.2.3-py3-none-any.whl (126 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 126.3/126.3 kB 1.1 MB/s eta 0:00:00 Collecting Events Downloading Events-0.5-py3-none-any.whl (6.8 kB) Collecting six Downloading six-1.16.0-py2.py3-none-any.whl (11 kB) Collecting pyasn1>=0.1.3 Downloading pyasn1-0.6.1-py3-none-any.whl (83 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.1/83.1 kB 1.1 MB/s eta 0:00:00 Collecting greenlet!=0.4.17 Downloading greenlet-3.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (643 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 643.3/643.3 kB 1.2 MB/s eta 0:00:00 Collecting h11>=0.8 Downloading h11-0.14.0-py3-none-any.whl (58 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.3/58.3 kB 1.1 MB/s eta 0:00:00 Collecting click>=7.0 Downloading click-8.1.7-py3-none-any.whl (97 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.9/97.9 kB 1.2 MB/s eta 0:00:00 Collecting pycparser Downloading pycparser-2.22-py3-none-any.whl (117 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 117.6/117.6 kB 1.1 MB/s eta 0:00:00 Collecting dnspython>=2.0.0 Downloading dnspython-2.6.1-py3-none-any.whl (307 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.7/307.7 kB 1.2 MB/s eta 0:00:00 Collecting typer>=0.12.3 Downloading typer-0.12.5-py3-none-any.whl (47 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.3/47.3 kB 1.0 MB/s eta 0:00:00 Collecting MarkupSafe>=2.0 Downloading MarkupSafe-2.1.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB) Collecting pathable<0.5.0,>=0.4.1 Downloading pathable-0.4.3-py3-none-any.whl (9.6 kB) Collecting rfc3339-validator Downloading rfc3339_validator-0.1.4-py2.py3-none-any.whl (3.5 kB) Collecting pydantic-core==2.23.4 Downloading pydantic_core-2.23.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 1.2 MB/s eta 0:00:00 Collecting annotated-types>=0.6.0 Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB) Collecting charset-normalizer<4,>=2 Downloading charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 142.1/142.1 kB 1.1 MB/s eta 0:00:00 Collecting exceptiongroup>=1.0.2 Downloading exceptiongroup-1.2.2-py3-none-any.whl (16 kB) Collecting uvicorn[standard]>=0.12.0 Downloading uvicorn-0.30.6-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 998.4 kB/s eta 0:00:00 Downloading uvicorn-0.30.5-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 1.6 MB/s eta 0:00:00 Downloading uvicorn-0.30.4-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 1.1 MB/s eta 0:00:00 Downloading uvicorn-0.30.3-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 1.0 MB/s eta 0:00:00 Downloading uvicorn-0.30.2-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.5/62.5 kB 1.1 MB/s eta 0:00:00 Downloading uvicorn-0.30.1-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.4/62.4 kB 1.0 MB/s eta 0:00:00 Downloading uvicorn-0.30.0-py3-none-any.whl (62 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.4/62.4 kB 994.5 kB/s eta 0:00:00 Downloading uvicorn-0.29.0-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 1.1 MB/s eta 0:00:00 Downloading uvicorn-0.28.1-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.5/60.5 kB 1.0 MB/s eta 0:00:00 Downloading uvicorn-0.28.0-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.6/60.6 kB 1.0 MB/s eta 0:00:00 Downloading uvicorn-0.27.1-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 1.1 MB/s eta 0:00:00 Downloading uvicorn-0.27.0.post1-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.7/60.7 kB 1.1 MB/s eta 0:00:00 Downloading uvicorn-0.27.0-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.6/60.6 kB 1.0 MB/s eta 0:00:00 Downloading uvicorn-0.26.0-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.5/60.5 kB 1.4 MB/s eta 0:00:00 Downloading uvicorn-0.25.0-py3-none-any.whl (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.3/60.3 kB 1.1 MB/s eta 0:00:00 Collecting python-dotenv>=0.13 Downloading python_dotenv-1.0.1-py3-none-any.whl (19 kB) Collecting watchfiles>=0.13 Downloading watchfiles-0.24.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (425 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 425.7/425.7 kB 1.2 MB/s eta 0:00:00 Collecting httptools>=0.5.0 Downloading httptools-0.6.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (341 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 341.4/341.4 kB 1.2 MB/s eta 0:00:00 Collecting websockets>=10.4 Downloading websockets-13.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (164 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.1/164.1 kB 1.2 MB/s eta 0:00:00 Collecting rich>=10.11.0 Downloading rich-13.9.1-py3-none-any.whl (242 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 242.1/242.1 kB 1.2 MB/s eta 0:00:00 Collecting shellingham>=1.3.0 Downloading shellingham-1.5.4-py2.py3-none-any.whl (9.8 kB) Collecting pygments<3.0.0,>=2.13.0 Downloading pygments-2.18.0-py3-none-any.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 1.2 MB/s eta 0:00:00 Collecting markdown-it-py>=2.2.0 Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 1.1 MB/s eta 0:00:00 Collecting mdurl~=0.1 Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB) Installing collected packages: pytz, mypy-extensions, Events, chardet, brotli, xmltodict, websockets, uvloop, uuid6, urllib3, typing-extensions, sniffio, six, shellingham, secure, rpds-py, PyYAML, python-multipart, python-json-logger, python-dotenv, PyJWT, pygments, pycparser, pyasn1, psutil, pathable, packaging, more-itertools, mdurl, MarkupSafe, lazy-object-proxy, inflection, idna, httptools, h11, greenlet, future, frozenlist, exceptiongroup, dnspython, defusedxml, Cython, content-size-limit-asgi, click, charset-normalizer, certifi, cachetools, attrs, async-timeout, annotated-types, werkzeug, uvicorn, SQLAlchemy, rsa, rfc3339-validator, requests, referencing, python-dateutil, pydantic-core, multidict, markdown-it-py, Jinja2, httpcore, gunicorn, email_validator, cffi, asgiref, anyio, aiosignal, yarl, watchfiles, starlette, rich, pydantic, opensearch-py, jsonschema-specifications, jsonschema-path, httpx, cryptography, typer, jsonschema, brotli-asgi, aiohttp, openapi-schema-validator, fastapi-cli, connexion, openapi-spec-validator, fastapi Running setup.py install for future: started Running setup.py install for future: finished with status 'done' Successfully installed Cython-0.29.36 Events-0.5 Jinja2-3.1.4 MarkupSafe-2.1.5 PyJWT-2.8.0 PyYAML-6.0.2 SQLAlchemy-2.0.23 aiohttp-3.9.5 aiosignal-1.3.1 annotated-types-0.7.0 anyio-4.6.0 asgiref-3.8.1 async-timeout-4.0.3 attrs-24.2.0 brotli-1.1.0 brotli-asgi-1.4.0 cachetools-4.1.0 certifi-2024.8.30 cffi-1.17.1 chardet-3.0.4 charset-normalizer-3.3.2 click-8.1.7 connexion-3.1.0 content-size-limit-asgi-0.1.5 cryptography-42.0.4 defusedxml-0.6.0 dnspython-2.6.1 email_validator-2.2.0 exceptiongroup-1.2.2 fastapi-0.111.1 fastapi-cli-0.0.5 frozenlist-1.4.1 future-0.18.3 greenlet-3.1.1 gunicorn-22.0.0 h11-0.14.0 httpcore-1.0.6 httptools-0.6.1 httpx-0.26.0 idna-3.10 inflection-0.5.1 jsonschema-4.20.0 jsonschema-path-0.3.3 jsonschema-specifications-2023.12.1 lazy-object-proxy-1.10.0 markdown-it-py-3.0.0 mdurl-0.1.2 more-itertools-8.2.0 multidict-6.1.0 mypy-extensions-0.4.3 openapi-schema-validator-0.6.2 openapi-spec-validator-0.7.1 opensearch-py-2.6.0 packaging-24.1 pathable-0.4.3 psutil-5.9.0 pyasn1-0.6.1 pycparser-2.22 pydantic-2.9.2 pydantic-core-2.23.4 pygments-2.18.0 python-dateutil-2.8.1 python-dotenv-1.0.1 python-json-logger-2.0.2 python-multipart-0.0.12 pytz-2020.1 referencing-0.35.1 requests-2.32.3 rfc3339-validator-0.1.4 rich-13.9.1 rpds-py-0.20.0 rsa-4.7.2 secure-0.3.0 shellingham-1.5.4 six-1.16.0 sniffio-1.3.1 starlette-0.37.2 typer-0.12.5 typing-extensions-4.12.2 urllib3-2.2.3 uuid6-2024.7.10 uvicorn-0.24.0.post1 uvloop-0.17.0 watchfiles-0.24.0 websockets-13.1 werkzeug-3.0.4 xmltodict-0.12.0 yarl-1.13.1 ```

The next step is testing the 5.0 code using the pre-installed with all the dependecies.

nico-stefani commented 2 months ago

Update

After manually installing the generated pre-installed Python. I've been able to start the wazuh services (cluster,api,comms_api) using it.

root@wazuh-master:/var/ossec# ll framework/python/
total 44
drwxr-x--- 1 root wazuh 4096 Oct  3 19:37 ./
drwxr-x--- 1 root wazuh 4096 Sep 20 19:12 ../
drwxr-xr-x 1 root root  4096 Oct  2 19:58 bin/
drwxr-xr-x 1 root root  4096 Oct  2 18:55 include/
drwxr-xr-x 1 root root  4096 Oct  2 18:55 lib/
drwxr-xr-x 1 root root  4096 Oct  2 18:55 share/
root@wazuh-master:/var/ossec# framework/python/bin/python3
Python 3.10.15 (v3.10.15-dirty:ffee63f, Oct  2 2024, 18:54:55) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
root@wazuh-master:/var/ossec# bin/wazuh-control info
WAZUH_VERSION="v5.0.0"
WAZUH_REVISION="50000"
WAZUH_TYPE="server"
root@wazuh-master:/var/ossec# bin/wazuh-control status
wazuh-clusterd is running...
wazuh-modulesd is running...
wazuh-monitord is running...
wazuh-logcollector is running...
wazuh-remoted is running...
wazuh-syscheckd is running...
wazuh-analysisd is running...
wazuh-maild not running...
wazuh-execd is running...
wazuh-db is running...
wazuh-authd is running...
wazuh-agentlessd not running...
wazuh-integratord not running...
wazuh-dbd not running...
wazuh-csyslogd not running...
wazuh-apid is running...
wazuh-comms-apid is running...
root@wazuh-master:/var/ossec# tail -f logs/cluster.log
2024/10/03 19:38:43 INFO: [Master] [Local integrity] Starting.
2024/10/03 19:38:43 INFO: [Master] [Local integrity] Finished in 0.002s. Calculated metadata of 3 files.
2024/10/03 19:38:51 INFO: [Master] [Local integrity] Starting.
2024/10/03 19:38:51 INFO: [Master] [Local integrity] Finished in 0.002s. Calculated metadata of 3 files.
2024/10/03 19:38:59 INFO: [Master] [Local integrity] Starting.
2024/10/03 19:38:59 INFO: [Master] [Local integrity] Finished in 0.002s. Calculated metadata of 3 files.
2024/10/03 19:39:07 INFO: [Master] [Local integrity] Starting.
2024/10/03 19:39:07 INFO: [Master] [Local integrity] Finished in 0.002s. Calculated metadata of 3 files.
2024/10/03 19:39:15 INFO: [Master] [Local integrity] Starting.
2024/10/03 19:39:15 INFO: [Master] [Local integrity] Finished in 0.002s. Calculated metadata of 3 files.
2024/10/03 19:39:23 INFO: [Master] [Local integrity] Starting.
2024/10/03 19:39:23 INFO: [Master] [Local integrity] Finished in 0.002s. Calculated metadata of 3 files.
root@wazuh-master:/var/ossec# tail -f logs/api.log
2024/10/03 18:37:23 INFO: Listening on 0.0.0.0:55000.
2024/10/03 18:37:23 INFO: Populating installation UID...
2024/10/03 18:37:23 INFO: Getting updates information...
2024/10/03 18:38:35 INFO: Shutdown wazuh-apid server.
2024/10/03 18:41:18 INFO: Checking RBAC database integrity...
2024/10/03 18:41:18 INFO: /var/ossec/api/configuration/security/rbac.db file was detected
2024/10/03 18:41:18 INFO: RBAC database integrity check finished successfully
2024/10/03 18:41:20 INFO: Listening on 0.0.0.0:55000.
2024/10/03 18:41:20 INFO: Getting installation UID...
2024/10/03 18:41:20 INFO: Getting updates information...
root@wazuh-master:/var/ossec# tail -f logs/comms_api.log
2024/10/03 18:37:17 INFO: Generated private key file in /var/ossec/api/configuration/ssl/server.crt
2024/10/03 18:37:17 INFO: Generated certificate file in /var/ossec/api/configuration/ssl/server.crt
2024/10/03 18:38:35 ERROR: Worker (pid:716) was sent SIGKILL! Perhaps out of memory?
2024/10/03 18:38:35 ERROR: Worker (pid:718) was sent SIGKILL! Perhaps out of memory?
2024/10/03 18:38:35 ERROR: Worker (pid:717) was sent SIGKILL! Perhaps out of memory?
2024/10/03 18:38:35 ERROR: Worker (pid:719) was sent SIGKILL! Perhaps out of memory?

The next step is start working on the workflows to generate the pre-installed in a automated way.

nico-stefani commented 1 month ago

Update

Testing the compatiblity with the different OS, I realize that I was linking wrong the sqlite library into the pre-installed python.

oot@0ed93ad40304:/# /var/wazuh/framework/python/bin/python3.10
Python 3.10.15 (v3.10.15-dirty:ffee63f, Oct  4 2024, 19:18:57) [GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/wazuh/framework/python/lib/python3.10/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/var/wazuh/framework/python/lib/python3.10/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: libsqlite3.so.0: cannot open shared object file: No such file or directory

I'll continue working on this topic.

nico-stefani commented 1 month ago

Update

I've been following an official Python guide for statically building the interpreted and trying different options but still without success.

./configure LDFLAGS="-L/usr/lib/sqlite3/lib/ -static -static-libgcc" CFLAGS="-static" CPPFLAGS="-I/usr/lib/sqlite3/include/ -static" LINKFORSHARED=" " \
    --prefix="/var/wazuh/framework/python" \
    --libdir="/var/wazuh/framework/python/lib" \
    --disable-shared \
    --enable-optimizations

make -j$(nproc) LDFLAGS="-static -static-libgcc" CPPFLAGS="-static" LINKFORSHARED=" "
make altinstall

Also, I found that it is possible to use the file Modules/Setup.local, with the desired configurations, in order to not modify the source code at Modules/Setup

# Edit this file for local setup changes

*static*

_contextvars _contextvarsmodule.c  # Context Variables
_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI _elementtree.c  # elementtree accelerator
_pickle -DPy_BUILD_CORE_MODULE _pickle.c    # pickle accelerator
_datetime _datetimemodule.c # datetime accelerator
_asyncio _asynciomodule.c  # Fast asyncio Future
_json -I$(srcdir)/Include/internal -DPy_BUILD_CORE_BUILTIN _json.c  # _json speedups

fcntl fcntlmodule.c # fcntl(2) and ioctl(2)
#spwd spwdmodule.c      # spwd(3)
grp grpmodule.c     # grp(3)
select selectmodule.c   # select(2); not on ancient System V

# Socket module helper for socket(2)
_socket socketmodule.c

OPENSSL=/usr/lib/openssl
# To statically link OpenSSL:
_ssl _ssl.c \
    -I$(OPENSSL)/include -L$(OPENSSL)/lib64 \
    -l:libssl.a -Wl,--exclude-libs,libssl.a \
    -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a
_hashlib _hashopenssl.c \
    -I$(OPENSSL)/include -L$(OPENSSL)/lib64 \
    -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a

_posixsubprocess  -DPy_BUILD_CORE_BUILTIN _posixsubprocess.c  # POSIX subprocess module helper

# The _md5 module implements the RSA Data Security, Inc. MD5
# Message-Digest Algorithm, described in RFC 1321.

_md5 md5module.c

# The _sha module implements the SHA checksum algorithms.
# (NIST's Secure Hash Algorithms.)
_sha1 sha1module.c
_sha256 sha256module.c -DPy_BUILD_CORE_BUILTIN
_sha512 sha512module.c -DPy_BUILD_CORE_BUILTIN
_sha3 _sha3/sha3module.c

# _blake module
_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c
nico-stefani commented 1 month ago

Update

I've been trying to build Python with SQLite statically but I couldn't achieve it.

We need to discuss how to proceed with the team.

nico-stefani commented 1 month ago

Partial Conclusion

Status of the build with static links

During this time we've trying to build the Python interpreter with static links of the libraries used during the compilation. Our focus were in OpenSSL and SQLite3. So far we achive the static linking of OpenSSL, as @fdalmaup mentions in https://github.com/wazuh/wazuh/issues/17454#issuecomment-2383257085. On the other hand we've facing some issues trying to statically link SQLite3. This is principally is because the internals of the python compilation process are different comparing it with the one for OpenSSL. So, cannot apply the same recipe to get it working.

At this point we are considering include SQLite3 as system dependency of the Wazuh server package. This is a topic that we need to discuss with the management.

During this investigation we discard other options like pyenv and pyinstaller.

Status of the dependecies

Regarding the depedencies we proposed a new way to handle the installation, taking in account that we'll distributing only the pre-installed from 5.0 and beyond. Also, we've cleaning the requirements.txt removing the wodles dependecies and the keeping only the direct dependecies of the framework and the API's.

Also, this set of depedencies was validated in https://github.com/wazuh/wazuh/issues/17454#issuecomment-2392192804 when we start up the Wazuh services with a interpreter prototype.

Pending Work

Still is pending

Next Steps

We decided to put this on hold to focus our efforts on #25790.

For the MVP stage we'll use a mix of the old pre-compiled, using the libraries wazuhext and the openssl, but installing the new set of dependencies directly with pip. Without the necessity to handle any wheel.