search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.14k stars 2.25k forks source link

Poetry lock does not include explicit sources for conditional packages #9537

Open Holt59 opened 2 months ago

Holt59 commented 2 months ago

Description

Following pyproject.toml from https://github.com/python-poetry/poetry/issues/6409 to allow user to install CPU or GPU versions of torch via poetry install -E cpu --with cpu or poetry install -E cuda --with cuda.

CUDA installation works fine, but CPU does not, throws a Repository "pytorch-cpu" does not exist.. When checking the poetry.lock file, the explicit source pytorch-cpu is missing and I cannot find a way to include it.

Workarounds

No workaround for now, I am completely stuck.

Poetry Installation Method

install.python-poetry.org

Operating System

Ubuntu 24.04

Poetry Version

1.8.3

Poetry Configuration

cache-dir = "/home/XXX/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = false
repositories.pytorch-cpu.url = "https://download.pytorch.org/whl/cpu"
repositories.pytorch-cuda.url = "https://download.pytorch.org/whl/cu118"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/XXX/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

[tool.poetry]
name = "pkg"
version = "0.1.0"
description = ""
authors = ["XXX <xxx@xxx.xxx>"]
readme = "README.md"

[[tool.poetry.source]]
name = "pytorch-cuda"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

[[tool.poetry.source]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

[tool.poetry.dependencies]
python = "^3.12"

[tool.poetry.group.cpu]
optional = true

[tool.poetry.group.cpu.dependencies]
torch = {version = "^2.3.1+cpu", extras = ["cpu"], source = "pytorch-cpu"}
torchaudio = {version = "^2.3.1+cpu", extras = ["cpu"], source = "pytorch-cpu"}
torchvision = {version = "^0.18.1+cpu", extras = ["cpu"], source = "pytorch-cpu"}

[tool.poetry.group.cuda]
optional = true

[tool.poetry.group.cuda.dependencies]
torch = {version = "^2.3.1+cu118", source = "pytorch-cuda", extras=["cuda"]}
torchaudio = {version = "^2.3.1+cu118", source = "pytorch-cuda", extras=["cuda"]}
torchvision = {version = "^0.18.1+cu118", source = "pytorch-cuda", extras=["cuda"]}

[tool.poetry.extras]
cuda = ["torch", "torchaudio", "torchvision"]
cpu = ["torch", "torchaudio", "torchvision"]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

Adding repository pytorch-cuda (https://download.pytorch.org/whl/cu118) and setting it as explicit
Adding repository pytorch-cpu (https://download.pytorch.org/whl/cpu) and setting it as explicit
Using virtualenv: /home/XXX/Desktop/stage-ines-londres/venv-cpu
Installing dependencies from lock file

Finding the necessary packages for the current system

  Stack trace:

  21  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/cleo/application.py:327 in run
       325│
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  20  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│

  19  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│

  18  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/cleo/application.py:473 in _run_command
       471│
       472│         if error is not None:
     → 473│             raise error
       474│
       475│         return terminate_event.exit_code

  17  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/cleo/application.py:457 in _run_command
       455│
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  16  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│
     → 117│         return self.execute(io) or 0
       118│
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  15  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/cleo/commands/command.py:61 in execute
        59│
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  14  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/console/commands/install.py:153 in handle
       151│         self.installer.verbose(self.io.is_verbose())
       152│
     → 153│         return_code = self.installer.run()
       154│
       155│         if return_code != 0:

  13  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│
     → 104│         return self._do_install()
       105│
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

  12  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/installation/installer.py:301 in _do_install
       299│
       300│         with solver.use_environment(self._env):
     → 301│             ops = solver.solve(use_latest=self._whitelist).calculate_operations(
       302│                 with_uninstalls=self._requires_synchronization,
       303│                 synchronize=self._requires_synchronization,

  11  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/puzzle/solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│

  10  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/puzzle/solver.py:154 in _solve
       152│
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│
       156│             packages = result.packages

   9  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│
     →  18│     return solver.solve()
        19│

   8  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│
       177│             return self._result()

   7  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:493 in _choose_package_version
       491│             return is_specific_marker, preference, -num_packages
       492│
     → 493│         dependency = min(unsatisfied, key=_get_min)
       494│
       495│         locked = self._provider.get_locked(dependency)

   6  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:480 in _get_min
       478│
       479│             num_packages = len(
     → 480│                 self._dependency_cache.search_for(
       481│                     dependency, self._solution.decision_level
       482│                 )

   5  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:112 in search_for
       110│         # We could always use dependency.without_features() here,
       111│         # but for performance reasons we only do it if necessary.
     → 112│         packages = self._search_for_cached(
       113│             dependency.without_features() if dependency.features else dependency, key
       114│         )

   4  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/version_solver.py:93 in _search_for
        91│         # additional results this time.
        92│         if not packages:
     →  93│             packages = self._provider.search_for(dependency)
        94│
        95│         return packages

   3  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/puzzle/provider.py:300 in search_for
       298│             return PackageCollection(dependency, packages)
       299│
     → 300│         packages = self._pool.find_packages(dependency)
       301│
       302│         packages.sort(

   2  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/repositories/repository_pool.py:212 in find_packages
       210│         repository_name = dependency.source_name
       211│         if repository_name:
     → 212│             return self.repository(repository_name).find_packages(dependency)
       213│
       214│         packages: list[Package] = []

   1  ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/repositories/repository_pool.py:136 in repository
       134│
       135│     def repository(self, name: str) -> Repository:
     → 136│         return self._get_prioritized_repository(name).repository
       137│
       138│     def get_priority(self, name: str) -> Priority:

  IndexError

  Repository "pytorch-cpu" does not exist.

  at ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/repositories/repository_pool.py:145 in _get_prioritized_repository
      141│     def _get_prioritized_repository(self, name: str) -> PrioritizedRepository:
      142│         name = name.lower()
      143│         if self.has_repository(name):
      144│             return self._repositories[name]
    → 145│         raise IndexError(f'Repository "{name}" does not exist.')
      146│
      147│     def add_repository(
      148│         self,
      149│         repository: Repository,
dimbleby commented 2 months ago

per https://github.com/python-poetry/poetry/issues/6409#issuecomment-1793440536, I wouldn't expect using a project's own extras as markers to work

this is at heart duplicate https://github.com/python-poetry/poetry-core/pull/613, which tries to make that work