search

wolfi-dev / os

Main package repository for production Wolfi images
Other
786 stars 212 forks source link

py3-parallel-ssh cannot be used - stuck on python3.11 #23756

Closed smoser closed 3 weeks ago

smoser commented 1 month ago

py3-parallel-ssh is currently unable to be used even though it can be built.

$ docker run --rm -it cgr.dev/chainguard/wolfi-base:latest
% apk add py3-parallel-ssh
fetch https://packages.wolfi.dev/os/x86_64/APKINDEX.tar.gz
(1/25) Installing libbz2-1 (1.0.8-r8)
(2/25) Installing libexpat1 (2.6.2-r1)
(3/25) Installing libffi (3.4.6-r3)
(4/25) Installing gdbm (1.24-r0)
(5/25) Installing xz (5.6.2-r0)
(6/25) Installing libgcc (13.3.0-r0)
(7/25) Installing libstdc++ (13.3.0-r0)
(8/25) Installing mpdecimal (4.0.0-r2)
(9/25) Installing ncurses-terminfo-base (6.5_p20240629-r0)
(10/25) Installing ncurses (6.5_p20240629-r0)
(11/25) Installing readline (8.2-r7)
(12/25) Installing sqlite-libs (3.45.1-r1)
(13/25) Installing python-3.12-base (3.12.4-r1)
(14/25) Installing python-3.12 (3.12.4-r1)
(15/25) Installing py3-pycparser (2.22-r1)
(16/25) Installing py3-cffi (1.16.0-r3)
(17/25) Installing py3-greenlet (3.0.3-r1)
(18/25) Installing py3.12-setuptools (70.3.0-r0)
(19/25) Installing py3-zope.event (5.0-r0)
(20/25) Installing py3-zope.interface (6.3-r0)
(21/25) Installing py3-gevent (24.2.1-r0)
(22/25) Installing python-3.11-base (3.11.9-r5)
(23/25) Installing py3-ssh-python (1.0.0-r1)
(24/25) Installing py3-ssh2-python (1.0.0-r1)
(25/25) Installing py3-parallel-ssh (2.12.0-r1)
OK: 170 MiB in 39 packages

$ python3.12 -c 'from pssh.clients import ParallelSSHClient'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.12/site-packages/pssh/clients/__init__.py", line 19, in <module>
    from .native.parallel import ParallelSSHClient
  File "/usr/lib/python3.12/site-packages/pssh/clients/native/__init__.py", line 19, in <module>
    from .parallel import ParallelSSHClient
  File "/usr/lib/python3.12/site-packages/pssh/clients/native/parallel.py", line 20, in <module>
    from .single import SSHClient
  File "/usr/lib/python3.12/site-packages/pssh/clients/native/single.py", line 24, in <module>
    from ssh2.error_codes import LIBSSH2_ERROR_EAGAIN
ModuleNotFoundError: No module named 'ssh2'

The above demonstrates the failure. Here is the "why". The package knows it depends directly on

    runtime:
      - py3-gevent
      - py3-ssh2-python
      - py3-ssh-python

But those dependencies are spread where some are installed in python3.11 path and others in python3.12

% pkgs="py3-pycparser py3-cffi py3-greenlet py3.12-setuptools 
  py3-zope.event py3-zope.interface py3-gevent py3-ssh-python 
  py3-ssh2-python py3-parallel-ssh"
% for p in $pkgs do
   echo == $p == ; apk info -L $p | grep .py$ | head -n 1; done
== py3-pycparser ==
usr/lib/python3.12/site-packages/pycparser/__init__.py
== py3-cffi ==
usr/lib/python3.12/site-packages/cffi/__init__.py
== py3-greenlet ==
usr/lib/python3.12/site-packages/greenlet/__init__.py
== py3.12-setuptools ==
usr/lib/python3.12/site-packages/_distutils_hack/__init__.py
== py3-zope.event ==
usr/lib/python3.12/site-packages/zope/event/__init__.py
== py3-zope.interface ==
usr/lib/python3.12/site-packages/zope/interface/__init__.py
== py3-gevent ==
usr/lib/python3.12/site-packages/gevent/__init__.py
== py3-ssh-python ==
usr/share/py3-ssh/.venv/lib/python3.11/site-packages/_distutils_hack/__init__.py
== py3-ssh2-python ==
usr/share/py3-ssh2/.venv/lib/python3.11/site-packages/_distutils_hack/__init__.py
== py3-parallel-ssh ==
usr/lib/python3.12/site-packages/pssh/__init__.p

The problem is:

smoser commented 1 month ago

The whole 'python3' migration thing (python3.11 -> python3.12) is weird. I had some thoughts on it that I wanted to write down.

  1. We recently changed the SCA in melange to automatically add a specific python-3.N dependency when building a package. This is correct. A python package installs .py/.pyc files into a specific python-3.X site-packages, you can't just use python3.11 and import requests if py3-requests installed its files into /usr/lib/python3.12/site-packages/requests.
  2. I've removed several of the 'python-3' manual dependencies (package.dependencies.runtime). because a.) they're not strictly necessary and b.) at a future point after 'python3.13' is available, they'll actually be wrong and would drag in python3.13 on install.
  3. The python3 symlink is useful during package build, but problematic. When you install a package it generally writes a shebang referencing whatever executable ran the installer. Using python3.12 -m installer pkg.whl will write #!/usr/bin/python3.12 as shebang while python3 -m installer.pkg will write #!/usr/bin/python3. But during package build, you don't want to have to call python3.12 explicitly as that will become wrong at some point in the future.
  4. similarly to 3, images should ship python3, as no end user should be burdened with having to know what python3.X version they're using.
smoser commented 3 weeks ago

@justinvreeland, did you want to do a pull request for your commit ?

justinvreeland commented 3 weeks ago

@smoser Looks like I might've messed up some metadata but it should be here: https://github.com/wolfi-dev/os/pull/26270