Open aignas opened 1 year ago
Hi @aignas, Could you please provide the OS and version of the Bazel that you were using? (What is the output of bazel info release
?)
System details:
$ uname -a
Darwin FA21050003 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64
$ bazel info
Starting local Bazel server and connecting to it...
bazel-bin: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/execroot/_main/bazel-out/darwin-fastbuild/bin
bazel-genfiles: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/execroot/_main/bazel-out/darwin-fastbuild/bin
bazel-testlogs: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/execroot/_main/bazel-out/darwin-fastbuild/testlogs
character-encoding: file.encoding = ISO-8859-1, defaultCharset = ISO-8859-1
command_log: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/command.log
committed-heap-size: 268MB
execution_root: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/execroot/_main
gc-count: 4
gc-time: 22ms
install_base: /var/tmp/_bazel_ignas.anikevicius/install/3b6d3b89ac6edd12cb24971ba5ddc98f
java-home: /private/var/tmp/_bazel_ignas.anikevicius/install/3b6d3b89ac6edd12cb24971ba5ddc98f/embedded_tools/jdk
java-runtime: OpenJDK Runtime Environment (build 11.0.6+10-LTS) by Azul Systems, Inc.
java-vm: OpenJDK 64-Bit Server VM (build 11.0.6+10-LTS, mixed mode) by Azul Systems, Inc.
max-heap-size: 4294MB
output_base: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7
output_path: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/execroot/_main/bazel-out
package_path: %workspace%
release: release 6.3.2
repository_cache: /var/tmp/_bazel_ignas.anikevicius/cache/repos/v1
server_log: /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/java.log.fa21050003.ignas.anikevicius.log.java.20230904-190321.53487
server_pid: 53487
used-heap-size: 39MB
workspace: /Users/ignas.anikevicius/src/github/aignas/bazel_downloader_repro
However, I have been also successful in reproducing it with last_green
.
Running the repro script on Linux results in:
+ cat README.md
+ grep '^\$'
+ sed 's/$ //g'
+ grep -v 'bazel info'
+ grep -v 'uname -a'
+ xargs '-I{}' -L1 bash -c '{}'
xargs: warning: options --replace and -L are mutually exclusive, ignoring previous --replace value
bazel: line 1: {}: command not found
bazel: line 1: {}: command not found
If I run the steps in the README manually, I get:
$ bazel clean --expunge --async && bazel fetch --enable_bzlmod @rules_python//python:defs.bzl
INFO: Invocation ID: d0cad29d-b790-4531-aa02-8dbe4c033c08
INFO: Starting clean.
INFO: Output base moved to /home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465_tmp_992010_70446071-fc4b-402a-bf96-b44107142182 for deletion
Starting local Bazel server and connecting to it...
WARNING: Download from https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.24.0/rules_python-0.24.0.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: company-artifactory.com
INFO: Repository rules_python~0.24.0 instantiated at:
<builtin>: in <toplevel>
Repository rule http_archive defined at:
/home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465/external/bazel_tools/tools/build_defs/repo/http.bzl:379:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_python~0.24.0':
Traceback (most recent call last):
File "/home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465/external/bazel_tools/tools/build_defs/repo/http.bzl", line 139, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.24.0/rules_python-0.24.0.tar.gz] to /home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465/external/rules_python~0.24.0/temp16680747550657347476/rules_python-0.24.0.tar.gz: Unknown host: company-artifactory.com
ERROR: <builtin>: fetching http_archive rule //:rules_python~0.24.0: Traceback (most recent call last):
File "/home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465/external/bazel_tools/tools/build_defs/repo/http.bzl", line 139, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.24.0/rules_python-0.24.0.tar.gz] to /home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465/external/rules_python~0.24.0/temp16680747550657347476/rules_python-0.24.0.tar.gz: Unknown host: company-artifactory.com
ERROR: no such package '@rules_python~0.24.0//python': java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.24.0/rules_python-0.24.0.tar.gz] to /home/fhenneke/.cache/bazel/_bazel_fhenneke/e3b6d2c81e73801aa6740a0b376be465/external/rules_python~0.24.0/temp16680747550657347476/rules_python-0.24.0.tar.gz: Unknown host: company-artifactory.com
Loading: 0 packages loaded
which looks just like the expected failure from the WORKSPACE case.
@fmeum, thanks for the comment, I also observe that my repro.sh
is behaving differently on Linux and Mac as my Linux machine works as expected. And thanks for the tip that xargs
is not working the same way on Linux and Mac. I have updated the script to work on both platforms.
Full logs for the failure pasted below for convenience:
$ ./repro.sh
+ cat README.md
+ grep '^\$'
+ sed 's/$ //g'
+ grep -v 'bazel info'
+ grep -v 'uname -a'
+ grep -v repro.sh
+ xargs '-I{}' bash -c '{}'
2023/09/05 08:56:44 Using unreleased version at commit 290fc80a5aae9dea06de52deed7098a5b8443f26
2023/09/05 08:56:44 Downloading https://storage.googleapis.com/bazel-builds/artifacts/macos/290fc80a5aae9dea06de52deed7098a5b8443f26/bazel...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
INFO: Starting clean.
INFO: Output base moved to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7_tmp_55065_4b41858a-6bf4-4fde-aa58-95c67c27cb53 for deletion
2023/09/05 08:57:02 Using unreleased version at commit 290fc80a5aae9dea06de52deed7098a5b8443f26
Starting local Bazel server and connecting to it...
WARNING: Download from https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: company-artifactory.com
INFO: Repository rules_python instantiated at:
/Users/ignas.anikevicius/src/github/aignas/bazel_downloader_repro/WORKSPACE:3:13: in <toplevel>
Repository rule http_archive defined at:
/private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/bazel_tools/tools/build_defs/repo/http.bzl:379:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_python':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/bazel_tools/tools/build_defs/repo/http.bzl", line 139, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz] to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/rules_python/temp1006954228130245263/rules_python-0.23.1.tar.gz: Unknown host: company-artifactory.com
ERROR: /Users/ignas.anikevicius/src/github/aignas/bazel_downloader_repro/WORKSPACE:3:13: fetching http_archive rule //external:rules_python: Traceback (most recent call last):
File "/private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/bazel_tools/tools/build_defs/repo/http.bzl", line 139, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz] to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/rules_python/temp1006954228130245263/rules_python-0.23.1.tar.gz: Unknown host: company-artifactory.com
ERROR: Error computing the main repository mapping: no such package '@rules_python//python': java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz] to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/rules_python/temp1006954228130245263/rules_python-0.23.1.tar.gz: Unknown host: company-artifactory.com
2023/09/05 08:57:06 Using unreleased version at commit 290fc80a5aae9dea06de52deed7098a5b8443f26
INFO: Starting clean.
INFO: Output base moved to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7_tmp_55076_1d48f5d8-228f-48a8-a2f0-b91cad83d961 for deletion
2023/09/05 08:57:07 Using unreleased version at commit 290fc80a5aae9dea06de52deed7098a5b8443f26
Starting local Bazel server and connecting to it...
INFO: All external dependencies fetched successfully.
Loading: 1 packages loaded
2023/09/05 08:57:15 Using unreleased version at commit 290fc80a5aae9dea06de52deed7098a5b8443f26
INFO: Starting clean.
INFO: Output base moved to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7_tmp_55106_34bd3c93-911e-4d8e-9ad5-4b5d5dd356f0 for deletion
Starting local Bazel server and connecting to it...
INFO: Repository rules_python instantiated at:
/Users/ignas.anikevicius/src/github/aignas/bazel_downloader_repro/WORKSPACE:3:13: in <toplevel>
Repository rule http_archive defined at:
/private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/bazel_tools/tools/build_defs/repo/http.bzl:372:31: in <toplevel>
WARNING: Download from https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException Unknown host: company-artifactory.com
ERROR: An error occurred during the fetch of repository 'rules_python':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz] to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/rules_python/temp2498886707789242059/rules_python-0.23.1.tar.gz: Unknown host: company-artifactory.com
ERROR: /Users/ignas.anikevicius/src/github/aignas/bazel_downloader_repro/WORKSPACE:3:13: fetching http_archive rule //external:rules_python: Traceback (most recent call last):
File "/private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/bazel_tools/tools/build_defs/repo/http.bzl", line 132, column 45, in _http_archive_impl
download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz] to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/rules_python/temp2498886707789242059/rules_python-0.23.1.tar.gz: Unknown host: company-artifactory.com
ERROR: Error computing the main repository mapping: no such package '@rules_python//python': java.io.IOException: Error downloading [https://company-artifactory.com/artifactory/github-releases/bazelbuild/rules_python/releases/download/0.23.1/rules_python-0.23.1.tar.gz] to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7/external/rules_python/temp2498886707789242059/rules_python-0.23.1.tar.gz: Unknown host: company-artifactory.com
2023/09/05 08:57:18 Using unreleased version at commit 290fc80a5aae9dea06de52deed7098a5b8443f26
Starting local Bazel server and connecting to it...
INFO: Starting clean.
INFO: Output base moved to /private/var/tmp/_bazel_ignas.anikevicius/2170a98a6de670b82d7ef712358291b7_tmp_55156_64f43e71-84eb-4081-ae54-c19dfbd2b7c2 for deletion
Starting local Bazel server and connecting to it...
INFO: All external dependencies fetched successfully.
Loading: 1 packages loaded
The failure is expected, right?
If you allow me to re-use this thread, I am seeing unexpected behavior with bzlmod + --experimental_downloader_config
.
If you have a config like
rewrite (.*) commondatastorage.googleapis.com/test-bucket-that-does-not-exist/$1
rewrite (.*) $1
(we use this to first try our own mirror before going to the original dependency location).
Without bzlmod, we see expected warnings like
WARNING: Download from https://commondatastorage.googleapis.com/test-bucket-that-does-not-exist/go.dev/dl/?mode=json&include=all failed: class java.io.FileNotFoundException GET returned 404 Not Found
but the build then succeeding because the original URL is used.
With bzlmod, it fails with
ERROR: Error computing the main repository mapping: module not found in registries: protobuf@3.19.6
i.e., it seems only the first generated URL is tried.
$ bazel info release
release 6.3.2
but I also see this on 7.1.1.
ERROR: Error computing the main repository mapping: module not found in registries: protobuf@3.19.6
Which registry are you using, this error likely happens before the warning you'd like to see.
ERROR: Error computing the main repository mapping: module not found in registries: protobuf@3.19.6
Which registry are you using, this error likely happens before the warning you'd like to see.
The default registry (BCR), I am not configuring that. I can also build fine with bzlmod without --experimental_downloader_config
.
Oh I see, that might be because you are also overriding the URL for fetching files from the BCR. If the BCR registry is not mirrored, then Bazel will fail to fetch and resolve dependencies. You probably need to update the config file to not override URLs that begins with https://bcr.bazel.build
.
Minimal reproduction https://github.com/aignas/bazel_downloader_repro See context in https://bazelbuild.slack.com/archives/CA31HN1T3/p1693462264909799
Right now it seems that the downloader config is not working as expected because I have whitelisted only
bcr.bazel.build
and my company's artifactory URL, butbzlmod
is pulling all of the dependencies without rewriting their URLs and the fetch of the dependency is successful even though it is accessed via a URL that should be rewritten and fail (because the URL is a dummy one).