search

bazelbuild / rules_python

Bazel Python Rules
https://rules-python.readthedocs.io
Apache License 2.0
520 stars 534 forks source link

Silence download 404 errors when using bazel downloader (`experimental_index_url` and `experimental_extra_index_urls`) #2100

Open dougthor42 opened 1 month ago

dougthor42 commented 1 month ago

🚀 feature request

Relevant Rules

pip.parse when using experimental_index_url and experimental_extra_index_urls

Description

Using the bazel downloader can result in a lot of warnings:

$ bazel test //src/pyle_xc/fab/package_tester:web_api_test
Starting local Bazel server and connecting to it...
WARNING: Download from https://[REDACTED]/simple/rsa/ failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://[REDACTED]/simple/uv/ failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://[REDACTED]/simple/h11/ failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://[REDACTED]/simple/ply/ failed: class java.io.FileNotFoundException GET returned 404 Not Found
WARNING: Download from https://[REDACTED]/simple/jq/ failed: class java.io.FileNotFoundException GET returned 404 Not Found

Can these be silenced in some way?

This may not be feasible/reasonable via rules_python because it's part of Bazel core.

Describe the solution you'd like

An attribute on pip.parse would be nice.

Describe alternatives you've considered

I'm investigating the bazel --logging flag and asking around.

aignas commented 1 month ago

It's not possible to silence it, but it is possible to explicitly pass the packages that should use a different index, then the error is not present.

See https://rules-python.readthedocs.io/en/latest/api/python/extensions/pip.html#pip.parse.experimental_index_url_overrides

It might be a good idea to raise a ticket to bazelbuild/bazel to allow silencing it via some attribute on https://bazel.build/rules/lib/builtins/module_ctx#download. The only way to silence this would be to reimplement the downloading of the Python index in some other language (e.g. Rust, go or Python) or to have a lock file support that has URLs in the lockfile itself, then we would not need to query PyPI at all.

Leaving this open for now because there is a solution, but not sure how high on the priority list the solution is.

dougthor42 commented 1 month ago

Thanks @aignas! I've opened https://github.com/bazelbuild/bazel/issues/23173 for such thing - we'll see what they say.

I'll see about using experimental_index_url_overrides. It may not be super feasible given how many custom packages we have, but it's worth a shot. I might have to write a script that keeps things in sync somehow...