Self-referencing optional dependencies check PyPi version instead of local `pyproject.toml`

[goffi-contrib]

Hello,

In my project I'm using optional dependencies for various features. It looks like that:

[project]
name = "libervia-backend"
…
[project.optional-dependencies]
SVG = [
    "CairoSVG",
]
GUI = [
    "PyQt6"
]
TUI = [
    "term-image ~= 0.7.1"
]
remote-control = [
    "cbor2"
]
conferences = [
    "autobahn ~= 23.6"
]
all = [
    "libervia-backend[SVG]",
    "libervia-backend[GUI]",
    "libervia-backend[TUI]",
    "libervia-backend[remote-control]",
    "libervia-backend[conferences]"
]
…
[tool.hatch.envs.default]
features = ["all"]

(the whole file is available at https://repos.goffi.org/libervia-backend/file/tip/pyproject.toml#l97)

But when I'm going into dev environment, hatch is looking for those dependencies in the PyPi verion of libervia-backend instead of looking in the definitions above:

…
ERROR: Cannot install cryptography>=41.0.1, libervia-backend[conferences]==0.8.0, oldmemo==1.0.0 and pyopenssl==24.1.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested cryptography>=41.0.1
    oldmemo 1.0.0 depends on cryptography>=3.3.2
    pyopenssl 24.1.0 depends on cryptography<43 and >=41.0.5
    libervia-backend[conferences] 0.8.0 depends on cryptography<3.5

(the 0.8.0 version is the one on PyPi and doesn't have such conferences optional dependencies).

[ofek]

Can you please explain how you came to the conclusion that it's looking at PyPI rather than locally? I'm trying to understand.

[goffi-contrib]

Hi, thanks for your reply.

It's looking for libervia-backend==0.8 which is the Pypi version, my local version is 0.9.0.dev0, and it says that libervia-backend depends on cryptography<3.5 which was the dependency of version 0.8 (the one on PyPi, cf. https://repos.goffi.org/libervia-backend/file/f97be63871e4/setup.py) while the current pyproject.toml uses cryptography >= 41.0.1). So it really smells like it's checking the PyPi version, but I may have missed something.

[ofek]

Can you please tell me a command that I could run to reproduce?

[goffi-contrib]

@ofek sorry for the late answer, I was busy at a dev sprint last week and I forgot about this issue.

On a machine with Mercurial installed, you can reproduce with this:

% hg clone https://repos.goffi.org/libervia-backend/
% cd libervia-backend
# replace pyproject.toml with a simplified version for testing
% cat """
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "libervia-backend"
description = ""
license = "AGPL-3.0-or-later"
requires-python = ">=3.10"
authors = [
    { name = "Libervia Dev Team", email = "contact@goffi.org" },
]
dependencies = [
    "cryptography >= 41.0.1",
]

[project.optional-dependencies]
conferences = [
    "autobahn ~= 23.6"
]

all = [
    "libervia-backend[conferences]"
]

[tool.hatch.envs.default]
# we need system-package for GPG
system-packages = true
features = ["all"]

[tool.hatch.envs.dev]
dependencies = [
]

[tool.hatch.version]
path = "libervia/backend/__init__.py"
""" > pyproject.toml

% hatch -v -e dev shell

And the result is:

% hatch -v -e dev shell
Finished checking dependencies
Requirement already satisfied: cryptography>=41.0.1 in /usr/lib/python3.12/site-packages (42.0.6)
Collecting libervia-backend[conferences]
  Using cached libervia_backend-0.8.0-py3-none-any.whl.metadata (2.1 kB)
Requirement already satisfied: cffi>=1.12 in /usr/lib/python3.12/site-packages (from cryptography>=41.0.1) (1.16.0)
WARNING: libervia-backend 0.8.0 does not provide the extra 'conferences'
Requirement already satisfied: babel<3 in /usr/lib/python3.12/site-packages (from libervia-backend[conferences]) (2.14.0)
Requirement already satisfied: dbus-python<1.3 in /home/goffi/.local/share/hatch/env/virtual/libervia-backend/Yb3zO-f1/dev/lib/python3.12/site-packages (from libervia-backend[conferences]) (1.2.18)
Requirement already satisfied: html2text<2020.2 in /home/goffi/.local/share/hatch/env/virtual/libervia-backend/Yb3zO-f1/dev/lib/python3.12/site-packages (from libervia-backend[conferences]) (2020.1.16)
Requirement already satisfied: jinja2>=2.10.3 in /usr/lib/python3.12/site-packages (from libervia-backend[conferences]) (3.1.4)
Requirement already satisfied: langid<2 in /home/goffi/.local/share/hatch/env/virtual/libervia-backend/Yb3zO-f1/dev/lib/python3.12/site-packages (from libervia-backend[conferences]) (1.1.6)
Requirement already satisfied: lxml>=3.1.0 in /home/goffi/.local/share/hatch/env/virtual/libervia-backend/Yb3zO-f1/dev/lib/python3.12/site-packages (from libervia-backend[conferences]) (5.2.1)
Requirement already satisfied: markdown>=3.0 in /usr/lib/python3.12/site-packages (from libervia-backend[conferences]) (3.6)
Requirement already satisfied: miniupnpc<2.1 in /home/goffi/.local/share/hatch/env/virtual/libervia-backend/Yb3zO-f1/dev/lib/python3.12/site-packages (from libervia-backend[conferences]) (2.0.2)
Requirement already satisfied: mutagen<2 in /usr/lib/python3.12/site-packages (from libervia-backend[conferences]) (1.47.0)
Requirement already satisfied: netifaces<0.12 in /usr/lib/python3.12/site-packages (from libervia-backend[conferences]) (0.11.0)
Requirement already satisfied: pillow>=6.0.0 in /usr/lib/python3.12/site-packages (from libervia-backend[conferences]) (10.4.0)
Requirement already satisfied: progressbar2<3.54 in /home/goffi/.local/share/hatch/env/virtual/libervia-backend/Yb3zO-f1/dev/lib/python3.12/site-packages (from libervia-backend[conferences]) (3.53.3)
INFO: pip is looking at multiple versions of libervia-backend[conferences] to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install cryptography>=41.0.1 and libervia-backend[conferences]==0.8.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested cryptography>=41.0.1
    libervia-backend[conferences] 0.8.0 depends on cryptography<3.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
Finished syncing dependencies

The cryptography<3.5 dependency comes from libervia-backend 0.8 which is the default version available on PyPi (https://pypi.org/project/libervia-backend/).