poetry selects wrong version for nightly PyTorch on OSX

PaperclipBadger commented 2 years ago

[x] I am on the latest Poetry version.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS Monterey 12.4
Poetry version: 1.2.0b3
Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/PaperclipBadger/59a1a7df41ce0af8725a694abf84ec0b

Issue

I'm trying to install the nightly build of PyTorch, so that I can use the new MPS acceleration.

Adding the PyTorch index as a source works (in 1.2.0b3, but not 1.1.14):

[[tool.poetry.source]]
name = "pytorch-nightly-cpu"
url = "https://download.pytorch.org/whl/nightly/cpu"
indexed = true
secondary = true

But poetry selects the 1.13.0.dev20220811+cpu version, which does not have an build for OSX:

% poetry add --allow-prereleases --source pytorch-nightly-cpu torch -vvv 
Loading configuration file /Users/PaperclipBadger/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/PaperclipBadger/Code/mypackage/poetry.toml
Adding repository pytorch-nightly-cpu (https://download.pytorch.org/whl/nightly/cpu) and setting it as secondary
Using virtualenv: /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage
Project environment contains an empty path in sys_path, ignoring.
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.pytorch.org
[urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
Source (pytorch-nightly-cpu): 120 packages found for torch *
Using version ^1.13.0.dev20220811+cpu for torch

Updating dependencies
Resolving dependencies...
   1: fact: mypackage is 0.1.0
   1: derived: mypackage
   1: fact: mypackage depends on torch (^1.13.0.dev20220811+cpu)
   1: selecting mypackage (0.1.0)
   1: derived: torch (>=1.13.0.dev20220811+cpu,<2.0.0)
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
Source (pytorch-nightly-cpu): 1 packages found for torch >=1.13.0.dev20220811+cpu,<2.0.0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl HTTP/1.1" 200 192738976
... (if left running, this will download every nightly cpu build without finding a match)

adding the dependency directly to pyproject.toml also doesn't work but gives a different error:

torch = {version = "==1.13.0.dev20220811", allow-prereleases = true, source = "pytorch-nightly-cpu"}

% poetry install -vvv
Loading configuration file /Users/PaperclipBadger/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/PaperclipBadger/Code/mypackage/poetry.toml
Adding repository pytorch-nightly-cpu (https://download.pytorch.org/whl/nightly/cpu) and setting it as secondary
Using virtualenv: /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage
Project environment contains an empty path in sys_path, ignoring.
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 1 skipped

  • Installing torch (1.13.0.dev20220810+cpu)
  • Installing typing-extensions (4.3.0): Skipped for the following reason: Already installed
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.pytorch.org
[urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 146418
Skipping wheel torch-1.13.0.dev20220810+cpu-cp310-cp310-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp310-cp310-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp311-cp311-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp37-cp37m-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp37-cp37m-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp38-cp38-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp38-cp38-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp39-cp39-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220810+cpu-cp39-cp39-win_amd64.whl as this is not supported by the current environment

  Stack trace:

  5  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  4  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  3  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:437 in _execute_install
      435│ 
      436│     def _execute_install(self, operation: Install | Update) -> int:
    → 437│         status_code = self._install(operation)
      438│ 
      439│         self._save_url_reference(operation)

  2  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:472 in _install
      470│             archive = self._download_link(operation, Link(package.source_url))
      471│         else:
    → 472│             archive = self._download(operation)
      473│ 
      474│         operation_message = self.get_operation_message(operation)

  1  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:609 in _download
      607│ 
      608│     def _download(self, operation: Install | Update) -> Path:
    → 609│         link = self._chooser.choose_for(operation.package)
      610│ 
      611│         return self._download_link(operation, link)

  RuntimeError

  Unable to find installation candidates for torch (1.13.0.dev20220810+cpu)

  at ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│

Looking at the PyTorch index, we see that indeed there are no builds tagged +cpu for OSX - the OSX builds do not have a local version specifier: https://download.pytorch.org/whl/nightly/torch/

torch-1.13.0.dev20220811+cpu-cp310-cp310-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp310-cp310-win_amd64.whl
torch-1.13.0.dev20220811+cpu-cp311-cp311-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp37-cp37m-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp37-cp37m-win_amd64.whl
torch-1.13.0.dev20220811+cpu-cp38-cp38-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp38-cp38-win_amd64.whl
torch-1.13.0.dev20220811+cpu-cp39-cp39-linux_x86_64.whl
torch-1.13.0.dev20220811+cpu-cp39-cp39-win_amd64.whl
torch-1.13.0.dev20220811-cp310-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp310-none-macosx_11_0_arm64.whl
torch-1.13.0.dev20220811-cp37-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp37-none-macosx_11_0_arm64.whl
torch-1.13.0.dev20220811-cp38-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp38-none-macosx_11_0_arm64.whl
torch-1.13.0.dev20220811-cp39-none-macosx_10_9_x86_64.whl
torch-1.13.0.dev20220811-cp39-none-macosx_11_0_arm64.whl

poetry should select 1.13.0.dev20220811, not 1.13.0.dev20220811+cpu. This is what pip does:

% pip install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cpu
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/nightly/cpu
Collecting torch
  Using cached https://download.pytorch.org/whl/nightly/cpu/torch-1.13.0.dev20220811-cp310-none-macosx_11_0_arm64.whl (50.6 MB)
Requirement already satisfied: typing-extensions in /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages (from torch) (4.3.0)
Installing collected packages: torch
Successfully installed torch-1.13.0.dev20220811

dimbleby commented 2 years ago

I think poetry behaviour is unlikely to change here.

You'll perhaps do better asking the pytorch project to use a consistent versioning scheme - it looks a lot like an accident that macosx is treated differently from the others and if so they'll surely welcome a bug report / fix.

neersighted commented 2 years ago

Closing this as it can be solved with multi-constraint versions, and it's external to Poetry.

PaperclipBadger commented 2 years ago

@neersighted can you elaborate on how to solve this with multi-constraint versions? Is this something I can do now as a user?

neersighted commented 2 years ago

You can add an additional constraint for macOS that depends on the macOS version number (and keep the +cpu tagged version for platforms that are not macOS).

PaperclipBadger commented 2 years ago

I tried that, but it doesn't work (assuming the constraint you meant was platform = "darwin" and not something more specific). Even though poetry correctly determines that mypackage depends on 1.13.0dev20220811 (no +cpu), it only downloads the +cpu binaries for win32 and linux.

I'm pretty sure darwin is the correct platform, even thought the torch binaries are have macosx_11_0 where I would expect the platform to be:

>>> sys.platform
'darwin'

torch = [
  { version = "==1.13.0.dev20220811,!=1.13.0.dev20220811+cpu", platform = "darwin", allow-prereleases = true, source = "pytorch-nightly-cpu" },
  { version = "==1.13.0.dev20220811", platform = "linux", allow-prereleases = true, source = "pytorch-nightly-cpu" }
]

$ poetry install -vvv
Loading configuration file /Users/PaperclipBadger/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/PaperclipBadger/Code/mypackage/poetry.toml
Adding repository pytorch-nightly-cpu (https://download.pytorch.org/whl/nightly/cpu) and setting it as secondary
Using virtualenv: /Users/PaperclipBadger/.pyenv/versions/3.10.5/envs/mypackage
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: mypackage is 0.1.0
   1: derived: mypackage
   0: Duplicate dependencies for torch
   0: Merging requirements for torch (==1.13.0.dev20220811)
   1: fact: mypackage depends on torch (==1.13.0.dev20220811,!=1.13.0.dev20220811+cpu)
   1: selecting mypackage (0.1.0)
   1: derived: torch (==1.13.0.dev20220811)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.pytorch.org
[urllib3.connectionpool] Starting new HTTPS connection (1): download.pytorch.org:443
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 151772
Source (pytorch-nightly-cpu): 2 packages found for torch 1.13.0.dev20220811
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 151772
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl HTTP/1.1" 304 0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-win_amd64.whl HTTP/1.1" 304 0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp311-cp311-linux_x86_64.whl HTTP/1.1" 304 0
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp37-cp37m-linux_x86_64.whl HTTP/1.1" 200 192787858
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp37-cp37m-win_amd64.whl HTTP/1.1" 200 163106888
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp38-cp38-linux_x86_64.whl HTTP/1.1" 200 192735786
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp38-cp38-win_amd64.whl HTTP/1.1" 200 163130031
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp39-cp39-linux_x86_64.whl HTTP/1.1" 200 192735643
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp39-cp39-win_amd64.whl HTTP/1.1" 200 163039062
Source (pytorch-nightly-cpu): Downloading wheel: torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch-1.13.0.dev20220811%2Bcpu-cp310-cp310-linux_x86_64.whl HTTP/1.1" 304 0
   1: fact: torch (1.13.0.dev20220811+cpu) depends on typing-extensions (*)
   1: selecting torch (1.13.0.dev20220811+cpu)
   1: derived: typing-extensions
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/typing-extensions/json HTTP/1.1" 304 0
Source (PyPI): 18 packages found for typing-extensions *
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/typing-extensions/ HTTP/1.1" 403 None
Source (pytorch-nightly-cpu): Authorization error accessing https://download.pytorch.org/whl/nightly/cpu/typing-extensions/
   1: selecting typing-extensions (4.3.0)
   1: Version solving took 72.352 seconds.
   1: Tried 1 solutions.

Writing lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 1 update, 0 removals, 1 skipped

  • Updating torch (1.13.0.dev20220811 -> 1.13.0.dev20220811+cpu)
  • Installing typing-extensions (4.3.0): Skipped for the following reason: Already installed
[urllib3.connectionpool] https://download.pytorch.org:443 "GET /whl/nightly/cpu/torch/ HTTP/1.1" 200 151772
Skipping wheel torch-1.13.0.dev20220811+cpu-cp310-cp310-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp310-cp310-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp311-cp311-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp37-cp37m-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp37-cp37m-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp38-cp38-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp38-cp38-win_amd64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp39-cp39-linux_x86_64.whl as this is not supported by the current environment
Skipping wheel torch-1.13.0.dev20220811+cpu-cp39-cp39-win_amd64.whl as this is not supported by the current environment

  Stack trace:

  6  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:245 in _execute_operation
      243│ 
      244│             try:
    → 245│                 result = self._do_execute_operation(operation)
      246│             except EnvCommandError as e:
      247│                 if e.e.returncode == -2:

  5  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:317 in _do_execute_operation
      315│             return 0
      316│ 
    → 317│         result: int = getattr(self, f"_execute_{method}")(operation)
      318│ 
      319│         if result != 0:

  4  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:444 in _execute_update
      442│ 
      443│     def _execute_update(self, operation: Install | Update) -> int:
    → 444│         status_code = self._update(operation)
      445│ 
      446│         self._save_url_reference(operation)

  3  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:483 in _update
      481│ 
      482│     def _update(self, operation: Install | Update) -> int:
    → 483│         return self._install(operation)
      484│ 
      485│     def _remove(self, operation: Uninstall) -> int:

  2  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:472 in _install
      470│             archive = self._download_link(operation, Link(package.source_url))
      471│         else:
    → 472│             archive = self._download(operation)
      473│ 
      474│         operation_message = self.get_operation_message(operation)

  1  ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/executor.py:609 in _download
      607│ 
      608│     def _download(self, operation: Install | Update) -> Path:
    → 609│         link = self._chooser.choose_for(operation.package)
      610│ 
      611│         return self._download_link(operation, link)

  RuntimeError

  Unable to find installation candidates for torch (1.13.0.dev20220811+cpu)

  at ~/.pyenv/versions/3.10.5/envs/mypackage/lib/python3.10/site-packages/poetry/installation/chooser.py:103 in choose_for
       99│ 
      100│             links.append(link)
      101│ 
      102│         if not links:
    → 103│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      104│ 
      105│         # Get the best link
      106│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      107│

neersighted commented 2 years ago

Sorry, I realize now that Poetry doesn't consider the two version numbers to be different in a multiple constraint entry. I would suggest solving this by using Poetry's normal resolution on non-macOS platforms and hardcoding direct URL dependencies to the macOS wheels using the correct platform markers. The long-term fix to allow proper resolution would be consistent versioning.

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry

poetry selects wrong version for nightly PyTorch on OSX #6150

Issue