Poetry doesn't (correctly) follow markers for non-PyPi Dependencies (at least w/ Pytorch)

BuildBackBuehler commented 7 months ago

Description

So I have been using Poetry for a couple of weeks and its still an improvement over Pipenv by a longshot but there seems to be an underlying issue (and perhaps it is actually entirely on Pytorch's end/their bad metadata, rather than some foundational Pip wheel downloader problem.

I am using a Macbook Pro M1. And these are the markers I'd like to use -- but they are never followed, particularly on Pytorch.org downloads. I've had better luck with Tensorflow, for example, but I only need 1 or 2 of their packages direct-from-source as opposed to the nigthlys avail. through Pytorch.org so I can't say for sure.

markers = "platform_machine == 'arm64' and platform_machine != 'x86_64' and platform_machine != 'amd64' and sys_platform != 'win32' and sys_platform != 'linux' and implementation_version != '*+cpu'"

I was attempting to debug, figuring maybe it was a hiccup with 1 marker, but I removed all markers and couldn't download the specified wheel. It refuses to update to anything but the +cpu-based packages and they are all Wins/Linux based.

Workarounds

Yeah...no workaround I know of. I see Anaconda has nightly packs avail -- donno if Poetry would be able to build 'em despite their Conda specificity? PyPi doesn't have a DB of the Nightlys AFAIK.

Only way I have gotten around it has been 1. remove Torch and point it to the correct wheel 2. use pip to circumvent the issue...this one is semi-successful. If anything, I find sometimes Poetry reverts to Torch 2.3.0, for example, then I can make things less painful by removing Torch via Pip and reinstalling the 2.4.0.devXXXXXX with Pip and that'll work until packages depending on Torch are needing to be added. Rinse and repeat.

Poetry Installation Method

other

Operating System

MacOSX 14.5

Poetry Version

1.8.1/2

Poetry Configuration

> poetry config --list
cache-dir = "/Users/zack/.home/Library/Caches/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
repositories.mlc.url = "https://mlc.ai/wheels"
repositories.torch.url = "https://download.pytorch.org/whl/nightly/cpu/"
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # $GIT_HOME/LLMLife/.venv
virtualenvs.prefer-active-python = "1"
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Using Poetry through Mise (Rust ASDF, basically).

Example pyproject.toml

No response

Poetry Runtime Logs

[PoetryLog.txt](https://github.com/python-poetry/poetry/files/15195603/PoetryLog.txt)

BuildBackBuehler commented 7 months ago

Also, don't know if there's a way to specify which package versioning is prioritized, but that would be exceptionally helpful in general. I know there is source-wise. But one of the annoyances that exists is Datasets doesn't recognize or has been set by Pytorch to look for a stable version of Torch. I don't care if Datasets is buggy/doesn't work as intended @ 2.19, whereas I need Torch @ 2.4.

dimbleby commented 7 months ago

You did not provide pyproject.toml so we can only try to guess at what you are doing

It may be that you are trying to add torch both as a direct-url dependency and also requiring that it be added via a specific repository. Don't do that: do one or the other.

BuildBackBuehler commented 7 months ago

You did not provide pyproject.toml so we can only try to guess at what you are doing

It may be that you are trying to add torch both as a direct-url dependency and also requiring that it be added via a specific repository. Don't do that: do one or the other.

Whoops, had included so much info, figured there's no way more would help heh. Attached it below.

Ignorant of how Poetry works behind the scenes, the direct URL install attempts were "phishing" -- hoping that if Poetry latched on and installed from the links it would refresh the indexed file-SHAs it had on-file with the SHAs of links/files with X/Y/Z endings (all due to having a new approach/markers it would look for).

And if the Pyproject changed, could always revert the PyProject.toml to its former repository-based state for Torch. pyproject copy.txt

dimbleby commented 7 months ago

Then this all is working as expected and should be closed as user error, thanks!

BuildBackBuehler commented 7 months ago

Then this all is working as expected and should be closed as user error, thanks!

Huh? I'm saying that was my own troubleshooting, the issue still exists/persists with or without that. Same goes for with/without my markers.

- Installing torch (2.4.0.dev20240505+cpu): Failed This is after deleting poetry.lock and deleting the torch source cache. I disabled my markers. As to say this is the default behavior for Poetry on my machine

dimbleby commented 7 months ago

we did this already, you have asked poetry to install torch both as a direct url dependency and also from a specific repository. it cannot do both at once, and telling you so is not a bug.

BuildBackBuehler commented 7 months ago

we did this already, you have asked poetry to install torch both as a direct url dependency and also from a specific repository. it cannot do both at once, and telling you so is not a bug.

This is what happened doing Poetry update (after removing poetry.lock, deleting the torch source cache & having removed Torch from Poetry + any info files, clean install), no direct URL intervention. The only thing that does work is by doing a direct URL dependency which is inconvenient for nightly builds.

dimbleby commented 7 months ago

I feel as though we are talking past each other, but I cannot tell what you are trying to report. You have provided a screenshot showing poetry behaving correctly, we have discussed how that behaviour is correct, I do not follow what the conversation is now about.

Most torch issues are in some way duplicate #6409, perhaps you will find whatever it is you are looking for there.

BuildBackBuehler commented 7 months ago

I feel as though we are talking past each other, but I cannot tell what you are trying to report. You have provided a screenshot showing poetry behaving correctly, we have discussed how that behaviour is correct, I do not follow what the conversation is now about.

Most torch issues are in some way duplicate #6409, perhaps you will find whatever it is you are looking for there.

Poetry works correctly for 1 of the 2 methods so I confirmed that for you (the URL dependency). What is not working as intended (despite attempts to reel in Poetry's problemed behavior with markers) was finding the befitting version for my platform.

Though I may have just figured it out. I supplied the URL that Pytorch.org suggests for using as their index which is https://download.pytorch.org/whl/nightly/cpu and this is what the poetry.lock has provided as options for the package.

[[package]]
name = "torch"
version = "2.4.0.dev20240505+cpu"
description = "Tensors and Dynamic neural networks in Python with strong GPU acceleration"
optional = false
python-versions = ">=3.8.0"
files = [
    {file = "torch-2.4.0.dev20240505+cpu-cp310-cp310-linux_x86_64.whl", hash = "sha256:0d1bd174d941891c87f39c831619b0584e33b3ea2ab7d5e3e40472f34fe2075b"},
    {file = "torch-2.4.0.dev20240505+cpu-cp310-cp310-win_amd64.whl", hash = "sha256:f0c685b54b752f515ad55293ea74262f6bd1d9b1821ebeb536f8305fddd27319"},
    {file = "torch-2.4.0.dev20240505+cpu-cp311-cp311-linux_x86_64.whl", hash = "sha256:56919eca31e66dfecfa33404df09b10be4cf4a4da6c05aece33b88b16e4087b4"},
    {file = "torch-2.4.0.dev20240505+cpu-cp311-cp311-win_amd64.whl", hash = "sha256:b18d0709bf54ae620fb566b43a3dd802cad758d0871c54c7fb5488cf098a6e7c"},
    {file = "torch-2.4.0.dev20240505+cpu-cp312-cp312-linux_x86_64.whl", hash = "sha256:59fdc455c2616dd48174b89f46cb85a4b1b61d6494d1bb8aaf98d9c065c1d891"},
    {file = "torch-2.4.0.dev20240505+cpu-cp312-cp312-win_amd64.whl", hash = "sha256:f5b3ac246af03a79e18de5fa723cde61e3d908352090143e6b9a3647ccaa56db"},
    {file = "torch-2.4.0.dev20240505+cpu-cp38-cp38-linux_x86_64.whl", hash = "sha256:2b9ea5c100e8ffba31e206c6392450e746c7fde1ae3b6c2a0699e5a387a2daa5"},
    {file = "torch-2.4.0.dev20240505+cpu-cp38-cp38-win_amd64.whl", hash = "sha256:3e8dc55467dd254b14cfe09337f5636c236d66b4c47e8b8a53d9652c2df766a0"},
    {file = "torch-2.4.0.dev20240505+cpu-cp39-cp39-linux_x86_64.whl", hash = "sha256:c3859bc26707a34019b730db89e76ebf3042aca38f3aadf114f989d6b9db486e"},
    {file = "torch-2.4.0.dev20240505+cpu-cp39-cp39-win_amd64.whl", hash = "sha256:b85bdc7c6e08bc374283d55573e7754eef852f65bf303cd5301629e2599a9eaf"},
]

As I've mentioned, it opts for "2.4.0.dev20240505+cpu" -- even though that is not an option for Macs. Even if I put "^2.4.0.dev20240505" it'll do +cpu only. I use the /cpu/ index because that's what Pytorch.org suggests and it is smaller list to look through than https://download.pytorch.org/whl/nightly/ (and to clarify /cpu/ index still has torch-2.4.0.dev20240501-cp39-none-macosx_11_0_arm64.whl, etc., etc.

I have a feeling that without the /cpu/ in the URL, it will work as intended...just really, really slowly. But perhaps my markers will work now. We will see when I try this with my other Poetry project. Decided to try sans /cpu/ without markers first for simplicity/pure testing which is taking forever.

Sadly nope. Now it is going for roc-m ones. I wish there was a whl/nightly/mac. AFAIK that doesn't exist.

For now I will stick to the URL dependency method and I'll use a script to trigger when I update Poetry dependencies, it will take care of changing the "20240505" part of the URL to keep up with the nightly updates unless there's a way to do that with Poetry directly in the TOML I'm unaware of.

dimbleby commented 7 months ago

Then this is indeed all known and duplicate, and all on torch:

inconsistent versions between macos and other distributions is eg https://github.com/pytorch/pytorch/issues/110004, sounds like you will want to open a new version of that
slow locking is caused by the torch repository not providing hashes. In https://github.com/pytorch/pytorch/issues/76557 they have on-off resolved that for their main packages, sounds like you will want to ask them to do so for nightly too

The good news is that per those issues torch seem willing to engage

Anyway there is nothing for poetry to do here, please close

github-actions[bot] commented 5 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry