search

wazuh / wazuh-virtual-machines

Wazuh - Virtual Machines (OVA and AMI)
https://wazuh.com/
GNU General Public License v2.0
0 stars 0 forks source link

Workflow Python install error #79

Open CarlosALgit opened 1 week ago

CarlosALgit commented 1 week ago

Description

While working on this issue, I encountered an error in the Install Ansible task. This error was seen today, but I did perform a test last week, and the error didn't exist.

Doing a bit of research I discovered that the error is due to the installed Python version.

Last week workflow run: It can be seen [here](https://github.com/wazuh/wazuh-virtual-machines/actions/runs/11295649994/job/31418835742). ``` Run sudo apt-get update && sudo apt install -y python3 && python3 -m pip install --user ansible-core==2.16 Get:1 file:/etc/apt/apt-mirrors.txt Mirrorlist [142 B] Get:6 https://packages.microsoft.com/repos/azure-cli jammy InRelease [3596 B] Get:7 https://packages.microsoft.com/ubuntu/22.04/prod jammy InRelease [3632 B] Hit:2 http://azure.archive.ubuntu.com/ubuntu jammy InRelease Get:3 http://azure.archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] Hit:4 http://azure.archive.ubuntu.com/ubuntu jammy-backports InRelease Get:5 http://azure.archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] Get:8 https://packages.microsoft.com/repos/azure-cli jammy/main amd64 Packages [1867 B] Get:9 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main arm64 Packages [43.8 kB] Get:10 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main armhf Packages [15.7 kB] Get:11 https://packages.microsoft.com/ubuntu/22.04/prod jammy/main amd64 Packages [176 kB] Get:12 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2072 kB] Get:13 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [359 kB] Get:14 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.9 kB] Get:15 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2511 kB] Get:16 http://azure.archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [433 kB] Get:17 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1132 kB] Get:18 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [264 kB] Get:19 http://azure.archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.3 kB] Get:20 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1854 kB] Get:21 http://azure.archive.ubuntu.com/ubuntu jammy-security/main Translation-en [300 kB] Get:22 http://azure.archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] Get:23 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2451 kB] Get:24 http://azure.archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [422 kB] Get:25 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [910 kB] Get:26 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [180 kB] Get:27 http://azure.archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.5 kB] Fetched 13.5 MB in 2s (5546 kB/s) Reading package lists... WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Reading package lists... Building dependency tree... Reading state information... python3 is already the newest version (3.10.6-1~22.04.1). 0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded. Collecting ansible-core==2.16 Downloading ansible_core-2.16.0-py3-none-any.whl (2.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 50.4 MB/s eta 0:00:00 Requirement already satisfied: jinja2>=3.0.0 in /usr/lib/python3/dist-packages (from ansible-core==2.16) (3.0.3) Collecting resolvelib<1.1.0,>=0.5.3 Downloading resolvelib-1.0.1-py2.py3-none-any.whl (17 kB) Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible-core==2.16) (3.4.8) Requirement already satisfied: PyYAML>=5.1 in /usr/lib/python3/dist-packages (from ansible-core==2.16) (5.4.1) Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from ansible-core==2.16) (24.1) Installing collected packages: resolvelib, ansible-core Successfully installed ansible-core-2.16.0 resolvelib-1.0.1 ```
Today workflow run: This run can be seen [here](https://github.com/wazuh/wazuh-virtual-machines/actions/runs/11322985194/job/31484727949). ``` Run sudo apt-get update && sudo apt install -y python3 && python3 -m pip install --user ansible-core==2.16 sudo apt-get update && sudo apt install -y python3 && python3 -m pip install --user ansible-core==2.16 shell: /usr/bin/bash -e {0} env: COMPOSITE_NAME: linux-amazon-2-ami-amd64 ALLOCATOR_PATH: /tmp/allocatorvm_ami PLAYBOOKS_PATH: /home/runner/work/wazuh-virtual-machines/wazuh-virtual-machines/ami/playbooks/ AWS_DEFAULT_REGION: us-east-1 AWS_REGION: us-east-1 AWS_ACCESS_KEY_ID: *** AWS_SECRET_ACCESS_KEY: *** AWS_SESSION_TOKEN: *** WAZUH_VERSION: 4.10.0 Hit:1 http://azure.archive.ubuntu.com/ubuntu noble InRelease Get:2 http://azure.archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB] Hit:3 http://azure.archive.ubuntu.com/ubuntu noble-backports InRelease Get:4 http://azure.archive.ubuntu.com/ubuntu noble-security InRelease [126 kB] Get:5 https://packages.microsoft.com/repos/azure-cli noble InRelease [3564 B] Get:6 https://packages.microsoft.com/ubuntu/24.04/prod noble InRelease [3600 B] Get:7 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [542 kB] Get:8 http://azure.archive.ubuntu.com/ubuntu noble-updates/main Translation-en [133 kB] Get:9 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 c-n-f Metadata [9048 B] Get:10 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [386 kB] Get:11 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe Translation-en [160 kB] Get:12 http://azure.archive.ubuntu.com/ubuntu noble-updates/universe amd64 c-n-f Metadata [15.0 kB] Get:13 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 Packages [384 kB] Get:14 http://azure.archive.ubuntu.com/ubuntu noble-security/main Translation-en [84.6 kB] Get:15 http://azure.archive.ubuntu.com/ubuntu noble-security/main amd64 c-n-f Metadata [4708 B] Get:16 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 Packages [278 kB] Get:17 http://azure.archive.ubuntu.com/ubuntu noble-security/universe Translation-en [117 kB] Get:18 http://azure.archive.ubuntu.com/ubuntu noble-security/universe amd64 c-n-f Metadata [10.4 kB] Get:19 https://packages.microsoft.com/repos/azure-cli noble/main amd64 Packages [754 B] Get:20 https://packages.microsoft.com/ubuntu/24.04/prod noble/main amd64 Packages [12.1 kB] Get:21 https://packages.microsoft.com/ubuntu/24.04/prod noble/main armhf Packages [5061 B] Get:22 https://packages.microsoft.com/ubuntu/24.04/prod noble/main arm64 Packages [8310 B] Fetched 2410 kB in 1s (3273 kB/s) Reading package lists... WARNING: apt does not have a stable CLI interface. Use with caution in scripts. Reading package lists... Building dependency tree... Reading state information... python3 is already the newest version (3.12.3-0ubuntu2). 0 upgraded, 0 newly installed, 0 to remove and 32 not upgraded. error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install. If you wish to install a non-Debian-packaged Python package, create a virtual environment using python3 -m venv path/to/venv. Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make sure you have python3-full installed. If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage a virtual environment for you. Make sure you have pipx installed. See /usr/share/doc/python3.12/README.venv for more information. note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification. Error: Process completed with exit code 1. ```

As it can be seen in the logs above, the Python version used has changed.

[!WARNING]
This behavior also affects the other workflows that install and use Python. I could replicate it in this run of the Installation Assistant test.

Tasks

[!NOTE]
Another workaround is to use the Python version that works (3.10). For that, there is a Github Action in the Marketplace called setup-python. We have to consider it as an option if the using of the Python venv gets complex.

CarlosALgit commented 1 week ago

Update Report

I've been testing some options. I've tested using pipx, installing the python3.10 version and finally using the --break-system-packages parameter. This last test using the --break-system-packages parameter is working but I don't know if it's the right way to fix this. We have to discuss it with the team, but, for now, it's the only one working.

CarlosALgit commented 1 week ago

Update Report

This issue goes On Hold due to we have to discuss the final implementation of the fix.

CarlosALgit commented 1 week ago

Update Report

I've implemented using the Python venv, so this fixes the error we were having. It worked for the AMI workflow here.

On the other hand, I couldn't get it working for the OVA workflow. I've performed various tests but none of the proposed solutions worked. Two examples using the venv and not using it in the Run Ansible playbook to generate the OVA step can be seen here and here.