Python module validation errors after clean install on Debian 12

dffffffff commented 1 year ago

The problem

Validation fails after a fresh installed on Debian 12

FAIL: Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=4.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0'] Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application '

Most but not all packages are installed in an externally-managed-environment

Output of ./validate.php

===========================================
Component | Version
--------- | -------
LibreNMS  | 23.5.0 (2023-05-19T15:32:19+00:00)
DB Schema | 2023_05_12_071412_devices_expand_timetaken_doubles (251)
PHP       | 8.2.5
Python    | 3.11.2
Database  | MariaDB 10.11.2-MariaDB-1
RRDTool   | 1.7.2
SNMP      | 5.9.3
===========================================

[OK]    Composer Version: 2.5.5
[OK]    Dependencies up-to-date.
[OK]    Database connection successful
[OK]    Database Schema is current
[OK]    SQL Server meets minimum requirements
[OK]    lower_case_table_names is enabled
[OK]    MySQL engine is optimal
[OK]    Database and column collations are correct
[OK]    Database schema correct
[OK]    MySQl and PHP time match
[OK]    Active pollers found
[OK]    Dispatcher Service not detected
[OK]    Locks are functional
[OK]    Python poller wrapper is polling
[OK]    Redis is unavailable
[FAIL]  Python3 module issue found: 'Required packages: ['PyMySQL!=1.0.0', 'python-dotenv', 'redis>=4.0', 'setuptools', 'psutil>=5.6.0', 'command_runner>=1.3.0']
Package not found: The 'command_runner>=1.3.0' distribution was not found and is required by the application
'
    [FIX]:
    pip3 install -r /opt/librenms/requirements.txt
[OK]    rrd_dir is writable
[OK]    rrdtool version ok

What was the last working version of LibreNMS?

No response

Anything in the logs that might be useful for us?


apt list --installed | grep python3

libpython3-dev/testing,now 3.11.2-1+b1 amd64 [installed,automatic]
libpython3-stdlib/testing,now 3.11.2-1+b1 amd64 [installed,automatic]
libpython3.11-dev/testing,now 3.11.2-6 amd64 [installed,automatic]
libpython3.11-minimal/testing,now 3.11.2-6 amd64 [installed,automatic]
libpython3.11-stdlib/testing,now 3.11.2-6 amd64 [installed,automatic]
libpython3.11/testing,now 3.11.2-6 amd64 [installed,automatic]
python3-apt/testing,now 2.5.3 amd64 [installed,automatic]
python3-async-timeout/testing,now 4.0.2-1 all [installed,automatic]
python3-certifi/testing,now 2022.9.24-1 all [installed,automatic]
python3-cffi-backend/testing,now 1.15.1-5+b1 amd64 [installed,automatic]
python3-chardet/testing,now 5.1.0+dfsg-2 all [installed,automatic]
python3-charset-normalizer/testing,now 3.0.1-2 all [installed,automatic]
python3-cryptography/testing,now 38.0.4-3 amd64 [installed,automatic]
python3-debconf/testing,now 1.5.82 all [installed,automatic]
python3-debian/testing,now 0.1.49 all [installed,automatic]
python3-debianbts/testing,now 4.0.1 all [installed,automatic]
python3-deprecated/testing,now 1.2.13-3 all [installed,automatic]
python3-dev/testing,now 3.11.2-1+b1 amd64 [installed,automatic]
python3-distutils/testing,now 3.11.2-3 all [installed,automatic]
python3-dotenv/testing,now 0.21.0-1 all [installed]
python3-httplib2/testing,now 0.20.4-3 all [installed,automatic]
python3-idna/testing,now 3.3-1 all [installed,automatic]
python3-lib2to3/testing,now 3.11.2-3 all [installed,automatic]
python3-minimal/testing,now 3.11.2-1+b1 amd64 [installed,automatic]
python3-packaging/testing,now 23.0-1 all [installed,automatic]
python3-pip/testing,now 23.0.1+dfsg-1 all [installed]
python3-pkg-resources/testing,now 66.1.1-1 all [installed,automatic]
python3-pycurl/testing,now 7.45.2-3 amd64 [installed,automatic]
python3-pymysql/testing,now 1.0.2-2 all [installed]
python3-pyparsing/testing,now 3.0.9-1 all [installed,automatic]
python3-pysimplesoap/testing,now 1.16.2-5 all [installed,automatic]
python3-redis/testing,now 4.3.4-3 all [installed]
python3-reportbug/testing,now 12.0.0 all [installed]
python3-requests/testing,now 2.28.1+dfsg-1 all [installed,automatic]
python3-setuptools/testing,now 66.1.1-1 all [installed]
python3-six/testing,now 1.16.0-4 all [installed,automatic]
python3-systemd/testing,now 235-1+b2 amd64 [installed]
python3-urllib3/testing,now 1.26.12-1 all [installed,automatic]
python3-wheel/testing,now 0.38.4-2 all [installed,automatic]
python3-wrapt/testing,now 1.14.1-2+b2 amd64 [installed,automatic]
python3.11-dev/testing,now 3.11.2-6 amd64 [installed,automatic]
python3.11-minimal/testing,now 3.11.2-6 amd64 [installed,automatic]
python3.11/testing,now 3.11.2-6 amd64 [installed,automatic]
python3/testing,now 3.11.2-1+b1 amd64 [installed,automatic]

PVasileff commented 1 year ago

pip3 install -U -r requirements.txt --break-system-packages fix the problem, but is it safe ?

slalomsk8er commented 1 year ago

Well, --break-system-packages makes it clear that it isn't save and now the user requested the explicit dangerous setup. If say, a update of a package brakes a OS script/tool, it can be serious or not but how would you know after running pip3 install -U -r requirements.txt --break-system-packages ? A proper solution would be to use a dedicated virtualenv for LibreNMS. This would also have the benefit of protecting LibreNMS from breakage caused by other pip3 runs. https://www.debian.org/releases/stable/amd64/release-notes/ch-information.en.html#python3-pep-668

murrant commented 1 year ago

As far as I'm concerned installing packages into the LibreNMS home directory will not break system packages, but the flag is still required there.

pip3 install --user -r requirements.txt --break-system-packages

I am not familiar with venv, so someone else will have to update docs and code to cope. And also take care not to break existing installs.

slalomsk8er commented 1 year ago

Are both --user and --break-system-packages needed? If packages are installed into the user context they shouldn't be able to break the system IMHO...

PVasileff commented 1 year ago

I have upgraded from debian 11 to debian 12.

After update Debian I have run ./validate.php and got same error.

To fix this - as root I run:

apt install python3-full
mv /usr/lib/python3.11/EXTERNALLY-MANAGED /tmp/

and as librenms I run only:

pip3 install -r /opt/librenms/requirements.txt

That fix the case.

murrant commented 1 year ago

@PVasileff that works, but it seems like a good way to break your system :D

ottorei commented 1 year ago

It should be possible to run it in a venv by using the venv's Python binary itself. But you would still need to be in the correct working directory. For example, if the venv was installed in /opt/librenms/venv, you could use the pip3 and python3 from /opt/librenms/venv/bin. If your working directory was /opt/librenms, you could just do ./venv/bin/pip3 -install -r requirements.txt. But obviously all Python scripts from this venv would need to be either run with sourcing the venv's activation script first or by directly calling the binaries from the venv.

philippludwig commented 9 months ago

I am facing this when I try to install librenms on a new system, following the official guide: https://docs.librenms.org/Installation/Install-LibreNMS

Now it’s unclear to me how to proceed.

mairie-varces commented 8 months ago

Just tested on Debian 12:

install following doc
validate.php fails on python module as described
as librenms user, in librenms install dir : python3 -m venv venv
source venv/bin/activate then pip3 install -r requirements.txt
edit /opt/librenms/.profile to set PATH to use venv binaries : export PATH=/opt/librenms/venv/bin:$PATH
resume documented install procedure : validate.php succeeds. Yay!

slalomsk8er commented 8 months ago

@mairie-varces This sounds to me like the correct way to do it.

mairie-varces commented 8 months ago

@slalomsk8er we're running librenms in a lab setup for now (trying it out!) and we're still running into surprises so my suggestion is not a fix yet, but it may help people work around the issue.

slalomsk8er commented 8 months ago

@mairie-varces I guess the path needs to be set in anything that isn't sourcing /opt/librenms/.profile but calling librenms python scripts - poller cronjobs, php-fpm, snmp-traps handler?

mairie-varces commented 8 months ago

The validation script succeeds when run from the shell but does not when called by php-fpm via /LibreNMS/Validations/Python.php.

The python validation script is called via a symfony Process function. Is this the case for every python script used by librenms? If so I have the fix!

After setting up the venv as described in my previous post, setup php-fpm to use the correct path by adding the following line to /etc/php/8.2/fpm/pool.d/librenms.conf : ENV[PATH] = /opt/librenms/venv/bin:/usr/local/bin:/usr/bin:/bin

As cron has a limited default path also add PATH=/opt/librenms/venv/bin:/usr/local/bin:/usr/bin:/bin at the start of /etc/cron.d/librenms.

@slalomsk8er are you aware of any other way to launch python programs in librenms? So far, with a bit of tinkering and grep, I have identified:

CLI utilities (example: validate.php)
python scripts called via symfony's Process (example: scripts/dynamic_check_requirements.py)
python scripts called via cron (example: discovery_wrapper.py)

slalomsk8er commented 8 months ago

@mairie-varces Maybe via PHP trough dist/librenms-scheduler.service but I'm not sure if a path set there will be carried though to any python programs.

Pelt10 commented 8 months ago

Hello, I'm currently testing a version with the python venv implementation.

The version is available at : Pelt10/librenms:python_venv I'm open to any review, help...

I can open an PR, if need, to continue discussion.

cheneraie commented 7 months ago

On Debian 12, only command-runner is missing. Other packages are system-wide available and installed. So this command fix the issue : pip3 install command-runner --break-system-packages I guess that the system package won't be broken because it doesn't exist.

librenms-bot commented 6 months ago

This issue has been mentioned on LibreNMS Community. There might be relevant details there:

https://community.librenms.org/t/looking-for-instructions-to-upgrade-debian-11-bullseye-os-and-librenms-v22-9-0-3-to-current-stable-version/23590/3

librenms-bot commented 1 month ago

This issue has been mentioned on LibreNMS Community. There might be relevant details there:

https://community.librenms.org/t/errors-validate-php/24970/2

librenms-bot commented 1 month ago

This issue has been mentioned on LibreNMS Community. There might be relevant details there:

https://community.librenms.org/t/errors-validate-php/24970/3

murrant commented 1 month ago

@Pelt10 I just switched snmpsim to use a venv in #15471 If you could change your code to be in line with that, it would be great. For example, it stores the venv in .python_venvs

slalomsk8er commented 2 weeks ago

.python_venvs

Why the s at the end?

murrant commented 2 weeks ago

@slalomsk8er because I assume we would want a different one for snmpsim vs the dispatcher.

slalomsk8er commented 2 weeks ago

@murrant ah, the s stands for snmpsim - I would have chosen something like .python_venv_s or .python_venv_snmpsim. Also why is it hidden, git?

BTW the steps from https://github.com/librenms/librenms/issues/15051#issuecomment-1855341079 & https://github.com/librenms/librenms/issues/15051#issuecomment-1857965364 but with .python_venvs worked well to upgrade my test system from Debian 11 to 12.

murrant commented 1 week ago

.python_venvs/snmpsim = snmpsim .python_venvs/dispatcher = dispatcher

librenms / librenms