python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.21k stars 2.25k forks source link

Only search source for designated platform and python #8871

Open JeffreyWardman opened 8 months ago

JeffreyWardman commented 8 months ago

The below will search the source for all wheels that match the version rather than restricting to the required platform and python version.

[tool.poetry.dependencies]
python = ">=3.10,<3.11.dev0"
torch = { version="2.2.0.dev20231213", source="torch", platform="linux", python="3.10" }

[[tool.poetry.source]]
name = "torch"
url = "https://download.pytorch.org/whl/nightly/cu121"
priority = "supplemental"

E.g. it also downloads: https://download.pytorch.org/whl/nightly/cu121/torch-2.2.0.dev20231213%2Bcu121-cp312-cp312-win_amd64.whl

Note the cp312 and win_amd64.

dimbleby commented 8 months ago

duplicate #7935

dimbleby commented 8 months ago

torch is particularly bad for this because they do not provide file hashes at the repository, so poetry feels the need to download all files itself to calculate those hashes.

https://github.com/pytorch/pytorch/issues/76557#issuecomment-1750463659 says this was fixed some time ago: but it does not seem to be the case at their nightly repository. Your fastest route to seeing improvement is likely to follow up with the torch folk to get that to happen

JeffreyWardman commented 8 months ago

Thanks for linking the preexisting issue. I missed it; my apologies. Surely there's a simple way to add a filter=* argument that will skip attempting to download files that don't contain that regex.

dimbleby commented 8 months ago

if you have a simple fix, please submit a merge request...