dougthor42 commented 2 weeks ago

🐞 bug report

Affected Rule

pip.parse

Is this a regression?

No. From what I can tell, it's always been present since experimental_index_url was added.

Description

Something wonky goes on when using a private index as the default pip index url. It seems like the credential helper is not working, but that may be a red herring.

🔬 Minimal Reproduction

This is fine:

pip.parse(
    experimental_extra_index_urls = ["https://[REDACTED]/simple"],
    experimental_index_url = "https://pypi.org/simple",
    hub_name = "pypi",
    python_version = "3.11",
    requirements_lock = "//:requirements.txt",
)

This fails:

pip.parse(
    experimental_extra_index_urls = ["https://pypi.org/simple"],
    experimental_index_url = "https://[REDACTED]/simple",
    hub_name = "pypi",
    python_version = "3.11",
    requirements_lock = "//:requirements.txt",
)

🔥 Exception or Error


ERROR: Traceback (most recent call last):                                                                              
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 472, column 52, in _pip_impl                                                                                                                                                                                                                                                                                                
                is_hub_reproducible = _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides, hub_group_map, simpleapi_cache)                                                                                                                                                                                                                                                                                                                                                     
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 187, column 50, in _create_whl_repos                                                                                                                                                                                                                                                                                        
                requirements_by_platform = parse_requirements(                                                                                                                                                                                 
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 348, column 37, in parse_requirements                                                                                                                                                                                                                                                                               
                whls, sdist = _add_dists(                                                                              
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 452, column 24, in _add_dists                                                                                                                                                                                                                                                                                       
                logger.warn("Could not find a whl or an sdist with sha256={}".format(sha256))                                                                                                                                                  
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 78, column 39, in lambda                                                                                                                                                                                                                                                                                                    
                warn = lambda message_cb: _log(0, "WARNING", message_cb),                                                                                                                                                                      
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 72, column 71, in _log                                                                                                                                                                                                                                                                                                      
                print("\nrules_python: {}: ".format(level.upper()), message_cb())  # buildifier: disable=print                                                                                                                                 
Error: 'string' object is not callable                                                                                 
ERROR: error evaluating module extension pip in @@rules_python~//python/extensions:pip.bzl                                                                                                                                                     
INFO: Elapsed time: 8.798s                                                                                             
INFO: 0 processes.                                                                                                     
ERROR: Build did NOT complete successfully                                                                             
Loading: 0 packages loaded                                                                                             
    currently loading:                                                                                                 
    Fetching module extension pip in @@rules_python~//python/extensions:pip.bzl; starting 8s

🌍 Your Environment

Operating System:

gLinux

Output of bazel version:

$ bazel --version
bazel 7.2.0rc1

We're using 7.2.0rc1 because of https://github.com/bazelbuild/bazel/issues/22664 which is actually a bug with jupyter/testpath that I haven't made a pip.override for yet.

Rules_python version:

Confirmed with:

0.33.1
0.33.2

Anything else relevant?

We use a credential helper to access the private index.

A different error happens with the latest rules_python commit at the time of writing (084b877c98b580839ceab2b071b02fc6768f3de6):

ERROR: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~'
ERROR: /usr/local/google/home/dthor/dev/pyle/BUILD.bazel:81:24: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' and referenced by '//:gazelle_python_manifest.test'
ERROR: Analysis of target '//:gazelle_python_manifest.test' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.454s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

This error happens for both "public pypi is main index" and "private index is main index" cases.

aignas commented 2 weeks ago

@dougthor42, could you see if main works differently? #2017 should help with debugging at least.

dougthor42 commented 2 weeks ago

could you see if main works differently?

Sorry I didn't put that front-and-center - it's hidden at the bottom of "Anything else relevant". When running on main, both cases result in:

ERROR: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~'
ERROR: /usr/local/google/home/dthor/dev/pyle/BUILD.bazel:81:24: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as '@com_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' and referenced by '//:gazelle_python_manifest.test'
ERROR: Analysis of target '//:gazelle_python_manifest.test' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.454s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

aignas commented 2 weeks ago

This does look like the gazelle plugin got removed during testing with main. It does not make sense that overriding rules_python would cause this. Unless you overrode both, gazelle and rules_python and we have a much bigger issue with the plugin.

I am slightly confused.

On the original topic though, the behaviour is different if your private index has some packages that have the same name as the public index, but the sha256 values in the private index are not the ones that your lock file is requesting.

This is my only idea why you would see any difference in behaviour.

On 4 July 2024 08:27:40 GMT+09:00, Douglas Thor @.***> wrote:

could you see if main works differently?

Sorry I didn't put that front-and-center - it's hidden at the bottom of "Anything else relevant". When running on main, both cases result in:

ERROR: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as ***@***.***_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~'
ERROR: /usr/local/google/home/dthor/dev/pyle/BUILD.bazel:81:24: no such package '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]//go/runfiles': The repository '@@[unknown repo 'com_github_bazelbuild_rules_go' requested from @@rules_python_gazelle_plugin~]' could not be resolved: No repository visible as ***@***.***_github_bazelbuild_rules_go' from repository '@@rules_python_gazelle_plugin~' and referenced by '//:gazelle_python_manifest.test'
ERROR: Analysis of target '//:gazelle_python_manifest.test' failed; build aborted: Analysis failed
INFO: Elapsed time: 9.454s
INFO: 0 processes.
ERROR: Build did NOT complete successfully

-- Reply to this email directly or view it on GitHub: https://github.com/bazelbuild/rules_python/issues/2033#issuecomment-2207494542 You are receiving this because you commented.

Message ID: @.***>

dougthor42 commented 2 weeks ago

I did play it fast and loose with the git_override when I was testing main. It's possible I f'd something up haha. We've got a long weekend coming up so you might not hear back until July 8th/9th.

aignas commented 2 weeks ago

Thanks, see you next week then. :)

On 4 July 2024 08:51:43 GMT+09:00, Douglas Thor @.***> wrote:

I did play it fast and loose with the git_override when I was testing main. It's possible I f'd something up haha. We've got a long weekend coming up so you might not hear back until July 8th/9th.

-- Reply to this email directly or view it on GitHub: https://github.com/bazelbuild/rules_python/issues/2033#issuecomment-2207551984 You are receiving this because you commented.

Message ID: @.***>

dougthor42 commented 1 week ago

OK here's more info:

TL;DR: It looks like commit b4b52fc89a58e6b7d5d0675b0661a09f932ec37e (#1917) is the culprit, and it was fixed in 11133b3e593243868a7a5926dab4fec5d455df40 (#2017).

It looks like (a) my git_override was mucked up - I'll have to check that later - and that (b) I didn't actually check main when I thought that I did. Sorry about that!

The rest of this comment is kept for my own sanity and reference. Feel free to ignore.

Procedure

To test, I chose a semi-random target and would run tests:

bazel test //src/pyle_xc/layout/scripts/drc:run_drc_test --cache_test_results=no

I cloned rules_python and used local_path_override to point my project to it. I then checked 0.33.1 through 0.33.2 but the issue persisted (meaning it wasn't already fixed in anything that's released).

I also went back to 0.32.0, when experimental_(extra_)?index_urls? were added and found that things worked. I also checked main (7bba79de34b6352001cb42b801245d0de33ce225 at the time of writing) and that did work.

So I git bisected backwards to find the issue and forward to find the fix.

Lastly, because I only realized at the end that I forgot to bazel clean --expunge, I did so and then tested the commit before the break, the breaking commit, the commit before the fix, and the fixing commit.

Results Table:

rules_python version	experimental _index_url	experimental _extra_index_urls	Result
0.32.0	pypi	private	OK
0.32.0	private	pypi	OK
b4b52fc89a58e6b7d5d0675b0661a09f932ec37e	pypi	private	OK
b4b52fc89a58e6b7d5d0675b0661a09f932ec37e	private	pypi	`Error: 'string' object is not callable`
0.33.1 (our current version)	pypi	private	OK
0.33.1 (our current version)	private	pypi	`Error: 'string' object is not callable`
0.33.2	pypi	private	OK
0.33.2	private	pypi	`Error: 'string' object is not callable`
7bba79de34b6352001cb42b801245d0de33ce225 (main)	pypi	private	OK
7bba79de34b6352001cb42b801245d0de33ce225 (main)	private	pypi	OK

How I patched:

# in MODULE.bazel
local_path_override(
    module_name = "rules_python",
    path = "/usr/local/google/home/dthor/dev/rules_python",
)
local_path_override(
    module_name = "rules_python_gazelle_plugin",
    path = "/usr/local/google/home/dthor/dev/rules_python/gazelle",
)

`Error: 'string' object is not callable'

ERROR: Traceback (most recent call last):
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 472, column 52, in _pip_impl
                is_hub_reproducible = _create_whl_repos(module_ctx, pip_attr, hub_whl_map, whl_overrides, hub_group_map, simpleapi_cache)
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/bzlmod/pip.bzl", line 187, column 50, in _create_whl_repos
                requirements_by_platform = parse_requirements(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 348, column 37, in parse_requirements
                whls, sdist = _add_dists(
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/parse_requirements.bzl", line 452, column 24, in _add_dists
                logger.warn("Could not find a whl or an sdist with sha256={}".format(sha256))
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 78, column 39, in lambda
                warn = lambda message_cb: _log(0, "WARNING", message_cb),
        File "/usr/local/google/home/dthor/.cache/bazel/_bazel_dthor/dbe74c4144b5c9a438d84a119652bef9/external/rules_python~/python/private/repo_utils.bzl", line 72, column 71, in _log
                print("\nrules_python: {}: ".format(level.upper()), message_cb())  # buildifier: disable=print
Error: 'string' object is not callable

dougthor42 commented 1 week ago

(a) my git_override was mucked up - I'll have to check that later

It was correct, but #2057 was preventing it from working correctly. :upside_down_face:

aignas commented 1 week ago

Thanks for the thorough investigation and the new ticket!

bazelbuild / rules_python

Error when `pip.parse.experimental_index_url` is a private index #2033

🐞 bug report

Affected Rule

Is this a regression?

Description

🔬 Minimal Reproduction

🔥 Exception or Error

🌍 Your Environment

Procedure

Results Table:

How I patched:

`Error: 'string' object is not callable'