search

pdm-project / pdm

A modern Python package and dependency manager supporting the latest PEP standards
https://pdm-project.org
MIT License
7.81k stars 386 forks source link

Bad python version in third party libraries cause resolution failure [pydantic] #3171

Closed evelina-gudauskayte closed 17 hours ago

evelina-gudauskayte commented 1 day ago

Make sure you run commands with -v flag before pasting the output.

Steps to reproduce

I have a dependency on pydantic~=2.7 in library_1. It gets resolved normally with pdm lock.

In library_2 I have a dependency on library_1. When doing a pdm lock in library_2 I get:

unearth.evaluator: Skipping link <Link https://files.pythonhosted.org/packages/f6/00/626c711ce35472ef35f57215372606ab1978511a290b04e086febdba4b12/pydantic-0.17-py36-none-any.whl (from https://pypi.org/simple/pydantic/)>: The wheel file pydantic-0.17-py36-none-any.whl is not compatible with the target environment (>=3.8,<3.12).
pdm.termui: Error occurs
Traceback (most recent call last):
  File "/home/python_venv/lib/python3.8/site-packages/pdm/termui.py", line 260, in logging
    yield logger
  File "/home/python_venv/lib/python3.8/site-packages/pdm/cli/actions.py", line 145, in do_lock
    mapping = _lock_for_env(
  File "/home/python_venv/lib/python3.8/site-packages/pdm/cli/actions.py", line 189, in _lock_for_env
    mapping, *_ = resolve(
  File "/home/python_venv/lib/python3.8/site-packages/pdm/resolver/core.py", line 43, in resolve
    result = resolver.resolve(requirements, max_rounds)
  File "/home/python_venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/home/python_venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 397, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/home/python_venv/lib/python3.8/site-packages/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/home/python_venv/lib/python3.8/site-packages/resolvelib/structs.py", line 127, in __bool__
    next(iter(self))
  File "/home/python_venv/lib/python3.8/site-packages/resolvelib/structs.py", line 136, in __iter__
    self._factory() if self._iterable is None else self._iterable
  File "/home/python_venv/lib/python3.8/site-packages/pdm/resolver/providers.py", line 230, in matches_gen
    candidates = self._find_candidates(reqs[0])
  File "/home/python_venv/lib/python3.8/site-packages/pdm/resolver/providers.py", line 200, in _find_candidates
    return self.repository.find_candidates(
  File "/home/python_venv/lib/python3.8/site-packages/pdm/models/repositories/base.py", line 180, in find_candidates
    cans = LazySequence(self._find_candidates(requirement, minimal_version=minimal_version))
  File "/home/python_venv/lib/python3.8/site-packages/pdm/models/repositories/pypi.py", line 68, in _find_candidates
    for c in finder.find_all_packages(req_name, allow_yanked=requirement.is_pinned)
  File "/home/python_venv/lib/python3.8/site-packages/unearth/finder.py", line 315, in find_all_packages
    self._find_packages(package_name, allow_yanked), hashes=hashes or {}
  File "/home/python_venv/lib/python3.8/site-packages/unearth/finder.py", line 295, in _find_packages
    return sorted(all_packages, key=self._sort_key, reverse=True)
  File "/home/python_venv/lib/python3.8/site-packages/unearth/evaluator.py", line 234, in evaluate_link
    self.check_wheel_tags(link.filename)
  File "/home/python_venv/lib/python3.8/site-packages/pdm/models/finder.py", line 56, in check_wheel_tags
    if self.env_spec.wheel_compatibility(filename) is None:
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/tags/tags.py", line 248, in wheel_compatibility
    return self.compatibility(
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/tags/tags.py", line 226, in compatibility
    python_compat = max(
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/tags/tags.py", line 228, in <genexpr>
    None, (self._evaluate_python(*comb) for comb in python_abi_combinations)
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/tags/tags.py", line 198, in _evaluate_python
    wheel_range = parse_version_specifier(f"=={major}.{minor}.*")
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/specifiers/__init__.py", line 118, in parse_version_specifier
    return from_specifierset(pkg_spec)
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/specifiers/__init__.py", line 27, in from_specifierset
    return functools.reduce(
  File "/home/python_venv/lib/python3.8/site-packages/dep_logic/specifiers/__init__.py", line 54, in _from_pkg_specifier
    min = Version(".".join([*version_parts, "0"]))
  File "/home/python_venv/lib/python3.8/site-packages/packaging/version.py", line 298, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
packaging.version.InvalidVersion: Invalid version: '3.6+.0'

Which is probably caused by a bad manifest in the pydantic.

Actual behavior

Raised packaging.version.InvalidVersion: Invalid version: '3.6+.0' error prevents finishing pdm lock because of old bad manifest.

Expected behavior

A warning with indication of skipping and bad python version in manifest instead of InvalidVersion error raise.

Environment Information

PDM version:
  2.18.2
Python Interpreter:
  /home/venv/bin/python (3.8)
Project Root:
  /mnt/e/work/projects/library_2
Local Packages:

INFO: Inside an active virtualenv /home/venv, reusing it.
Set env var PDM_IGNORE_ACTIVE_VENV to ignore it.
{
  "implementation_name": "cpython",
  "implementation_version": "3.8.18",
  "os_name": "posix",
  "platform_machine": "x86_64",
  "platform_release": "5.15.153.1-microsoft-standard-WSL2",
  "platform_system": "Linux",
  "platform_version": "#1 SMP Fri Mar 29 23:14:13 UTC 2024",
  "python_full_version": "3.8.18",
  "platform_python_implementation": "CPython",
  "python_version": "3.8",
  "sys_platform": "linux"
}
frostming commented 1 day ago

What are the versions of dep-logic and packaging respectively in your environment? You can see this by running pdm self list. I also work on projects depending on pydantic and never encounter problem

evelina-gudauskayte commented 18 hours ago

I also do not get this problem when pydantic is required, although that happens when I require a package that requires pydantic. Here us the pdm self list.

Library 2 pdm self list ``` Deprecated 1.2.14 Python @deprecated decorator to deprecate old python classes, functions or methods. GitPython 3.1.43 GitPython is a Python library used to interact with Git repositories Jinja2 3.1.4 A very fast and expressive template engine. MarkupSafe 2.1.5 Safely add untrusted strings to HTML/XML markup. PyYAML 6.0.2 YAML parser and emitter for Python Pygments 2.18.0 Pygments is a syntax highlighting package written in Python. altgraph 0.17.3 Python graph (network) package annotated-types 0.7.0 Reusable constraint types to use with typing.Annotated anyio 4.4.0 High level compatibility layer for multiple asynchronous event loop implementations asttokens 2.4.1 Annotate AST trees with source code positions backcall 0.2.0 Specifications for callback functions passed in to an API blinker 1.8.2 Fast, simple object-to-object and broadcast signaling cachetools 5.5.0 Extensible memoizing collections and decorators certifi 2024.8.30 Python package for providing Mozilla's CA Bundle. cfgv 3.4.0 Validate configuration and produce human readable error messages. charset-normalizer 3.3.2 The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet. comm 0.2.2 Jupyter Python Comm implementation, for usage in ipykernel, xeus-python etc. contourpy 1.1.1 Python library for calculating contours of 2D quadrilateral grids cycler 0.12.1 Composable style cycles debugpy 1.8.5 An implementation of the Debug Adapter Protocol for Python decorator 5.1.1 Decorators for Humans dep-logic 0.4.6 Python dependency specifications supporting logical operations distlib 0.3.8 Distribution utilities docstring_parser 0.16 Parse Python docstrings in reST, Google and Numpydoc format exceptiongroup 1.2.2 Backport of PEP 654 (exception groups) executing 2.1.0 Get the currently executing AST node of a frame, and other information filelock 3.16.1 A platform independent file lock. findpython 0.6.1 A utility to find python versions on your system fonttools 4.53.1 Tools to manipulate font files fsspec 2024.9.0 File-system specification gitdb 4.0.11 Git Object Database giturlparse 0.12.0 A Git URL parsing module (supports parsing and rewriting) google-api-core 2.19.2 Google API client core library google-auth 2.34.0 Google Authentication Library google-cloud-appengine-logging 1.4.5 Google Cloud Appengine Logging API client library google-cloud-audit-log 0.3.0 Google Cloud Audit Protos google-cloud-core 2.4.1 Google Cloud API client core library google-cloud-logging 3.11.2 Stackdriver Logging API client library google-cloud-storage 2.18.2 Google Cloud Storage API client library google-crc32c 1.5.0 A python wrapper of the C library 'Google CRC32C' google-resumable-media 2.7.2 Utilities for Google Media Downloads and Resumable Uploads googleapis-common-protos 1.65.0 Common protobufs used in Google APIs grpc-google-iam-v1 0.13.1 IAM API client library grpcio 1.66.1 HTTP/2-based RPC framework grpcio-status 1.66.1 Status proto mapping for gRPC h11 0.14.0 A pure-Python, bring-your-own-I/O implementation of HTTP/1.1 hishel 0.0.30 Persistent cache implementation for httpx and httpcore httpcore 1.0.5 A minimal low-level HTTP client. httpx 0.27.2 The next generation HTTP client. identify 2.6.1 File identification library for Python idna 3.10 Internationalized Domain Names in Applications (IDNA) importlib_metadata 8.5.0 Read metadata from Python packages importlib_resources 6.4.5 Read resources from Python packages iniconfig 2.0.0 brain-dead simple config-ini parsing installer 0.7.0 A library for installing Python wheels. ipykernel 6.29.5 IPython Kernel for Jupyter ipython 8.12.3 IPython: Productive Interactive Computing jedi 0.19.1 An autocompletion tool for Python that can be used for text editors. jupyter_client 8.6.3 Jupyter protocol implementation and client libraries jupyter_core 5.7.2 Jupyter core package. A base package on which Jupyter projects rely. kiwisolver 1.4.7 A fast implementation of the Cassowary constraint solver markdown-it-py 3.0.0 Python port of markdown-it. Markdown parsing, done right! matplotlib 3.7.2 Python plotting package matplotlib-inline 0.1.7 Inline Matplotlib backend for Jupyter mdurl 0.1.2 Markdown URL utilities mpmath 1.3.0 Python library for arbitrary-precision floating-point arithmetic msgpack 1.1.0 MessagePack serializer mypy-extensions 1.0.0 Type system extensions for programs checked with the mypy type checker. nest-asyncio 1.6.0 Patch asyncio to allow nested event loops networkx 3.1 Python package for creating and manipulating graphs and networks nodeenv 1.9.1 Node.js virtual environment builder numpy 1.24.4 Fundamental package for array computing in Python nvidia-cublas-cu12 12.1.3.1 CUBLAS native runtime libraries nvidia-cuda-cupti-cu12 12.1.105 CUDA profiling tools runtime libs. nvidia-cuda-nvrtc-cu12 12.1.105 NVRTC native runtime libraries nvidia-cuda-runtime-cu12 12.1.105 CUDA Runtime native Libraries nvidia-cudnn-cu12 9.1.0.70 cuDNN runtime libraries nvidia-cufft-cu12 11.0.2.54 CUFFT native runtime libraries nvidia-curand-cu12 10.3.2.106 CURAND native runtime libraries nvidia-cusolver-cu12 11.4.5.107 CUDA solver native runtime libraries nvidia-cusparse-cu12 12.1.0.106 CUSPARSE native runtime libraries nvidia-nccl-cu12 2.20.5 NVIDIA Collective Communication Library (NCCL) Runtime nvidia-nvjitlink-cu12 12.6.68 Nvidia JIT LTO Library nvidia-nvtx-cu12 12.1.105 NVIDIA Tools Extension opencv-python 4.8.0.76 Wrapper package for OpenCV python bindings. opencv-python-headless 4.8.0.76 Wrapper package for OpenCV python bindings. opentelemetry-api 1.27.0 OpenTelemetry Python API packaging 20.9 Core utilities for Python packages parso 0.8.4 A Python Parser patool 1.12 portable archive file manager pbs-installer 2024.9.9 Installer for Python Build Standalone pdm 2.18.2 A modern Python package and dependency manager supporting the latest PEP standards pdm-backend 2.3.3 The build backend used by PDM that supports latest packaging standards pep440-version-utils 0.3.0 Utilities to deal with pep440 versioning pexpect 4.9.0 Pexpect allows easy control of interactive console applications. pickleshare 0.7.5 Tiny 'shelve'-like database with concurrency support pillow 10.4.0 Python Imaging Library (Fork) pip 22.3.1 The PyPA recommended tool for installing Python packages. platformdirs 4.3.6 A small Python package for determining appropriate platform-specific dirs, e.g. a `user data dir`. pluggy 1.5.0 plugin and hook calling mechanisms for python pre-commit 3.5.0 A framework for managing and maintaining multi-language pre-commit hooks. prompt_toolkit 3.0.47 Library for building powerful interactive command lines in Python proto-plus 1.24.0 Beautiful, Pythonic protocol buffers. protobuf 5.28.1 psutil 6.0.0 Cross-platform lib for process and system monitoring in Python. ptyprocess 0.7.0 Run a subprocess in a pseudo terminal pure_eval 0.2.3 Safely evaluate AST nodes without side effects pyasn1 0.6.1 Pure-Python implementation of ASN.1 types and DER/BER/CER codecs (X.208) pyasn1_modules 0.4.1 A collection of ASN.1-based protocols modules pydantic 2.9.2 Data validation using Python type hints pydantic-settings 2.5.2 Settings management using Pydantic pydantic_core 2.23.4 Core functionality for Pydantic validation and serialization pydicom 2.4.4 A pure Python package for reading and writing DICOM data pyinstaller 5.13.2 PyInstaller bundles a Python application and all its dependencies into a single package. pyinstaller-hooks-contrib 2023.8 Community maintained hooks for PyInstaller pylibjpeg 2.0.1 A Python framework for decoding JPEG and decoding/encoding DICOM RLE data, with a focus on supporting pydicom pylibjpeg-libjpeg 2.2.0 A Python wrapper for libjpeg, with a focus on use as a plugin for for pylibjpeg pylibjpeg-openjpeg 2.3.0 A Python wrapper for openjpeg, with a focus on use as a plugin for for pylibjpeg pylibjpeg-rle 2.0.0 "Python bindings for a fast RLE decoder/encoder, with a focus on ""use as a plugin for pylibjpeg" pyparsing 3.0.9 pyparsing module - Classes and methods to define and execute parsing grammars pyproject_hooks 1.1.0 Wrappers to call pyproject.toml-based build backend hooks. pytest 8.3.3 pytest: simple powerful testing with Python python-dateutil 2.9.0.post0 Extensions to the standard Python datetime module python-dotenv 1.0.1 Read key-value pairs from a .env file and set them as environment variables python-gdcm 3.0.24.1 Grassroots DICOM runtime libraries pyzmq 26.2.0 Python bindings for 0MQ **library_1** 2.0.0 Python common library **library_2** 0.0.1.dev33+g67b61e0.d20240918 DICOM library1 **internal-packaging** 1.0.0 Shared utils for PDM-based packaging for internal purposes requests 2.32.3 Python HTTP for Humans. resolvelib 1.0.1 Resolve abstract dependencies into concrete ones rich 13.8.1 Render rich text, tables, progress bars, syntax highlighting, markdown and more to the terminal rsa 4.9 Pure-Python RSA implementation scipy 1.10.1 Fundamental algorithms for scientific computing in Python setuptools 65.5.1 Easily download, build, install, upgrade, and uninstall Python packages shellingham 1.5.4 Tool to Detect Surrounding Shell six 1.16.0 Python 2 and 3 compatibility utilities smmap 5.0.1 A pure Python implementation of a sliding window memory map manager sniffio 1.3.1 Sniff out which async library your code is running under socksio 1.0.0 Sans-I/O implementation of SOCKS4, SOCKS4A, and SOCKS5. stack-data 0.6.3 Extract data from python stack frames and tracebacks for informative displays sympy 1.13.2 Computer algebra system (CAS) in Python tomli 2.0.1 A lil' TOML parser tomlkit 0.13.2 Style preserving TOML library torch 2.4.1 Tensors and Dynamic neural networks in Python with strong GPU acceleration torchvision 0.19.1 image and video datasets and models for torch deep learning tornado 6.4.1 Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. tqdm 4.66.5 Fast, Extensible Progress Meter traitlets 5.14.3 Traitlets Python configuration system triton 3.0.0 A language and compiler for custom Deep Learning operations typed-argument-parser 1.10.1 Typed Argument Parser typing-inspect 0.9.0 Runtime inspection utilities for typing module. typing_extensions 4.12.2 Backported and Experimental Type Hints for Python 3.8+ unearth 0.17.2 A utility to fetch and download python packages urllib3 2.2.3 HTTP library with thread-safe connection pooling, file post, and more. virtualenv 20.26.5 Virtual Python Environment builder wcwidth 0.2.13 Measures the displayed width of unicode strings in a terminal wheel 0.38.4 A built-package format for Python wrapt 1.16.0 Module for decorators, wrappers and monkey patching. zipp 3.20.2 Backport of pathlib-compatible object wrapper for zip files ```
frostming commented 17 hours ago

packaging 20.9 Core utilities for Python packages

This is too old and not compatible with dep-logic: https://github.com/pdm-project/dep-logic/blob/9989d64fb93e33e258260848ca2ea3e4adf7dcf1/pyproject.toml#L8

Please upgrade the version of packaging and this issue can be solved

I also noticed that your packages also show up in this list. It is a bad practice since it will likely cause dependency conflicts. It's recommended to use a standalone environment for PDM and only install PDM dependencies and plugins into it. Don't install PDM into the same venv used by your project.