search

jd / pifpaf

Python fixtures and daemon managing tools for functional testing
Apache License 2.0
193 stars 39 forks source link

pifpaf has a dependency on distutils (and current releases on pkg_resources as well) that are not installed by default on python 3.12 and above #176

Closed zzzeek closed 1 month ago

zzzeek commented 1 month ago

pifpaf imports distutils and current release versions also import pkg_resources which was removed here (but it's still on current pypi release 3.1.5)

both of these tools are no longer included in python virtual environments:

https://docs.python.org/3/whatsnew/3.12.html

gh-95299: Do not pre-install setuptools in virtual environments created with venv. This means that distutils, setuptools, pkg_resources, and easy_install will no longer available by default; to access these run pip install setuptools in the activated virtual environment.

the simple solution is to add setuptools as a dependency. alternatively, remove the use of distutils

I can't easily demonstrate the distutils failure installing from github master due to #177 , unless you modify setup.cfg to also install drivers, it then looks like this:

(.venv) [classic@framework pifpaf:add_valkey]$ pip install .
Processing /home/classic/dev/pifpaf
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting daiquiri (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached daiquiri-3.2.5.1-py3-none-any.whl.metadata (1.6 kB)
Collecting click (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting jinja2 (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting fixtures (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached fixtures-4.1.0-py3-none-any.whl.metadata (21 kB)
Collecting packaging (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB)
Collecting psutil (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting xattr (from pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached xattr-1.1.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting python-json-logger (from daiquiri->pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached python_json_logger-2.0.7-py3-none-any.whl.metadata (6.5 kB)
Collecting pbr>=5.7.0 (from fixtures->pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached pbr-6.0.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting MarkupSafe>=2.0 (from jinja2->pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting cffi>=1.16.0 (from xattr->pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached cffi-1.17.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.16.0->xattr->pifpaf==3.1.6.dev47+ge7251a1.d20240808)
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached daiquiri-3.2.5.1-py3-none-any.whl (19 kB)
Using cached fixtures-4.1.0-py3-none-any.whl (64 kB)
Using cached jinja2-3.1.4-py3-none-any.whl (133 kB)
Using cached packaging-24.1-py3-none-any.whl (53 kB)
Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (290 kB)
Using cached xattr-1.1.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (39 kB)
Using cached cffi-1.17.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (479 kB)
Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Using cached pbr-6.0.0-py2.py3-none-any.whl (107 kB)
Using cached python_json_logger-2.0.7-py3-none-any.whl (8.1 kB)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Building wheels for collected packages: pifpaf
  Building wheel for pifpaf (pyproject.toml) ... done
  Created wheel for pifpaf: filename=pifpaf-3.1.6.dev47+ge7251a1.d20240808-py3-none-any.whl size=61472 sha256=4646f6c6d5e10e7b6fc5c7d0a96164a316be1f0e9772578c51168701b5fb055d
  Stored in directory: /tmp/pip-ephem-wheel-cache-2nk31oob/wheels/6a/5c/ed/c42abf20db70490f731c7550eee38a8793ca115412ab3a65fb
Successfully built pifpaf
Installing collected packages: python-json-logger, pycparser, psutil, pbr, packaging, MarkupSafe, click, jinja2, fixtures, daiquiri, cffi, xattr, pifpaf
Successfully installed MarkupSafe-2.1.5 cffi-1.17.0 click-8.1.7 daiquiri-3.2.5.1 fixtures-4.1.0 jinja2-3.1.4 packaging-24.1 pbr-6.0.0 pifpaf-3.1.6.dev47+ge7251a1.d20240808 psutil-6.0.0 pycparser-2.22 python-json-logger-2.0.7 xattr-1.1.0

[notice] A new release of pip is available: 24.1.2 -> 24.2
[notice] To update, run: pip install --upgrade pip
(.venv) [classic@framework pifpaf:add_valkey]$ cd ~/tmp/
(.venv) [classic@framework tmp]$ pifpaf run redis
CRITICAL [root] Traceback (most recent call last):
  File "/home/classic/tmp/.venv/bin/pifpaf", line 8, in <module>
    sys.exit(run_main())
             ^^^^^^^^^^
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/pifpaf/__main__.py", line 299, in run_main
    return main.main(standalone_mode=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/click/core.py", line 1682, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/click/core.py", line 1729, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/pifpaf/__main__.py", line 133, in get_command
    plugin = [e for e in DAEMONS if e.name == name][0].load()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/importlib/metadata/__init__.py", line 205, in load
    module = import_module(match.group('module'))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/home/classic/tmp/.venv/lib/python3.12/site-packages/pifpaf/drivers/__init__.py", line 24, in <module>
    from distutils import spawn
ModuleNotFoundError: No module named 'distutils'

an install from pypi fails on pkg_resources instead, we have to have setuptools in our tox dependencies to work around this:

[classic@framework tmp]$ python -m venv .venv
[classic@framework tmp]$ .venv/bin/pip install pifpaf
Collecting pifpaf
  Using cached pifpaf-3.1.5-py3-none-any.whl.metadata (7.0 kB)
Collecting daiquiri (from pifpaf)
  Using cached daiquiri-3.2.5.1-py3-none-any.whl.metadata (1.6 kB)
Collecting click (from pifpaf)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting pbr (from pifpaf)
  Using cached pbr-6.0.0-py2.py3-none-any.whl.metadata (1.3 kB)
Collecting jinja2 (from pifpaf)
  Using cached jinja2-3.1.4-py3-none-any.whl.metadata (2.6 kB)
Collecting fixtures (from pifpaf)
  Using cached fixtures-4.1.0-py3-none-any.whl.metadata (21 kB)
Collecting psutil (from pifpaf)
  Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (21 kB)
Collecting xattr (from pifpaf)
  Using cached xattr-1.1.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)
Collecting python-json-logger (from daiquiri->pifpaf)
  Using cached python_json_logger-2.0.7-py3-none-any.whl.metadata (6.5 kB)
Collecting MarkupSafe>=2.0 (from jinja2->pifpaf)
  Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting cffi>=1.16.0 (from xattr->pifpaf)
  Using cached cffi-1.17.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
Collecting pycparser (from cffi>=1.16.0->xattr->pifpaf)
  Using cached pycparser-2.22-py3-none-any.whl.metadata (943 bytes)
Using cached pifpaf-3.1.5-py3-none-any.whl (62 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached daiquiri-3.2.5.1-py3-none-any.whl (19 kB)
Using cached fixtures-4.1.0-py3-none-any.whl (64 kB)
Using cached pbr-6.0.0-py2.py3-none-any.whl (107 kB)
Using cached jinja2-3.1.4-py3-none-any.whl (133 kB)
Using cached psutil-6.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (290 kB)
Using cached xattr-1.1.0-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (39 kB)
Using cached cffi-1.17.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (479 kB)
Using cached MarkupSafe-2.1.5-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB)
Using cached python_json_logger-2.0.7-py3-none-any.whl (8.1 kB)
Using cached pycparser-2.22-py3-none-any.whl (117 kB)
Installing collected packages: python-json-logger, pycparser, psutil, pbr, MarkupSafe, click, jinja2, fixtures, daiquiri, cffi, xattr, pifpaf
Successfully installed MarkupSafe-2.1.5 cffi-1.17.0 click-8.1.7 daiquiri-3.2.5.1 fixtures-4.1.0 jinja2-3.1.4 pbr-6.0.0 pifpaf-3.1.5 psutil-6.0.0 pycparser-2.22 python-json-logger-2.0.7 xattr-1.1.0

[notice] A new release of pip is available: 23.3.2 -> 24.2
[notice] To update, run: python3.12 -m pip install --upgrade pip
[classic@framework tmp]$ .venv/bin/pifpaf run redis
Traceback (most recent call last):
  File "/home/classic/tmp/.venv/bin/pifpaf", line 5, in <module>
    from pifpaf.__main__ import run_main
  File "/home/classic/tmp/.venv/lib64/python3.12/site-packages/pifpaf/__main__.py", line 31, in <module>
    import pkg_resources
ModuleNotFoundError: No module named 'pkg_resources'
tobias-urdin commented 1 month ago

attempt to remove distutils in https://github.com/jd/pifpaf/pull/178 – then a new release should be cut