bazelbuild / rules_python

Bazel Python Rules
https://rules-python.readthedocs.io
Apache License 2.0
530 stars 540 forks source link

pip.parse doesn't fetch anything with conflicting markers #2337

Open keith opened 1 week ago

keith commented 1 week ago

If I have a requirements.in like this:

--extra-index-url https://download.pytorch.org/whl/cpu
torch==2.4.1; platform_machine != "x86_64"
torch==2.4.1+cpu; platform_machine == "x86_64"

Which I use to generate a universal requirements.txt with uv using: uv pip compile --universal requirements.in -o requirements_lock.txt --emit-index-url --no-strip-markers, I end up with multiple entries for torch in that lockfile:

SNIP
torch==2.4.1+cpu ; platform_machine == 'x86_64'
    # via -r requirements.in
torch==2.4.1 ; platform_machine != 'x86_64'
    # via -r requirements.in
SNIP

When I attempt to pull those in with:

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pip",
    python_version = "3.11",
    requirements_lock = "//:requirements_lock.txt",
)
use_repo(pip, "pip")

and use them with:

load("@pip//:requirements.bzl", "requirement")

alias(
    name = "foo",
    actual = requirement("torch"),
)

It fails with:

ERROR: no such package '@@rules_python~~pip~pip//torch': BUILD file not found in directory 'torch' of external repository @@rules_python~~pip~pip. Add a BUILD file to a directory to mark it as a package.
ERROR: /tmp/pyrepro/BUILD.bazel:3:6: no such package '@@rules_python~~pip~pip//torch': BUILD file not found in directory 'torch' of external repository @@rules_python~~pip~pip. Add a BUILD file to a directory to mark it as a package. and referenced by '//:foo'

If I remove the non-matching entry from the lockfile it works, so I assume it's the nature of having 2 conflicting requirements even though they are disambiguated by the marker.

To repro in this project: repro.zip

Run bazel fetch ...

aignas commented 1 week ago

I think the reason for this surprising behaviour is https://github.com/bazelbuild/rules_python/blob/7d4b8a559c59339f6916f1aa6358132194842537/python/private/pypi/parse_requirements.bzl#L88

We would have to have a fix there and ensure that it works with env markers.