search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.25k stars 1.26k forks source link

Spurious build errors (e.g. error downloading jdk) due to mirror.bazel.build #15740

Closed jwnimmer-tri closed 1 year ago

jwnimmer-tri commented 3 years ago

The frequency of builds dying due to https://github.com/bazelbuild/bazel/issues/12010 has increased in recent days, most particularly due to https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz.

jwnimmer-tri commented 3 years ago

We could improve this in CI by pre-populating the repository_cache, or inserting a local proxy server, or any other number of fixes, but that wouldn't help local users.

It's also dumb that we're downloading JDKs and Android tools and all of that junk that we don't need in the first place. I'm going to see how easy it might be to turn off some downloads that we don't actually need.

jwnimmer-tri commented 3 years ago

The full list of mirror files as reported by bazel sync is as follows:

https://mirror.bazel.build/bazel_android_tools/android_tools_pkg-0.23.0.tar.gz
https://mirror.bazel.build/bazel_coverage_output_generator/releases/coverage_output_generator-v2.5.zip
https://mirror.bazel.build/bazel_java_tools/releases/javac11/v10.6/java_tools_javac11_darwin-v10.6.zip
https://mirror.bazel.build/bazel_java_tools/releases/javac11/v10.6/java_tools_javac11_linux-v10.6.zip
https://mirror.bazel.build/bazel_java_tools/releases/javac11/v10.6/java_tools_javac11_windows-v10.6.zip
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu11.45.27-ca-jdk11.0.10-macosx_aarch64.tar.gz
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu14.28.21-ca-jdk14.0.1-linux_x64.tar.gz
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu14.28.21-ca-jdk14.0.1-macosx_x64.tar.gz
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu14.28.21-ca-jdk14.0.1-win_x64.zip
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu15.27.17-ca-jdk15.0.0-linux_x64.tar.gz
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu15.27.17-ca-jdk15.0.0-macosx_x64.tar.gz
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu15.27.17-ca-jdk15.0.0-win_x64.zip
https://mirror.bazel.build/cdn.azul.com/zulu/bin/zulu15.29.15-ca-jdk15.0.2-macosx_aarch64.tar.gz
https://mirror.bazel.build/github.com/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.7+10/OpenJDK11U-jdk_s390x_linux_hotspot_11.0.7_10.tar.gz
https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/b1c40e1de81913a3c40e5948f78719c28152486d.zip
https://mirror.bazel.build/github.com/bazelbuild/rules_java/archive/7cf3cefd652008d0a64a419c34c13bdca6c8f178.zip
https://mirror.bazel.build/github.com/bazelbuild/rules_proto/archive/7e4afce6fe62dbff0a4a03450143146f9f2d7488.tar.gz
https://mirror.bazel.build/openjdk/AdoptOpenJDK/openjdk11-binaries/releases/download/jdk-11.0.7+10/OpenJDK11U-jdk_ppc64le_linux_hotspot_11.0.7_10.tar.gz
https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz
https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-macosx_x64.tar.gz
https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-win_x64.zip
https://mirror.bazel.build/openjdk/azul-zulu11.37.48-ca-jdk11.0.6/zulu11.37.48-ca-jdk11.0.6-linux_aarch64.tar.gz

We are probably not using all of those during a bazel build //.... I'm working on a narrower report.

Edit: Here's the list of files used during the default build:

https://mirror.bazel.build/bazel_coverage_output_generator/releases/coverage_output_generator-v2.5.zip
https://mirror.bazel.build/bazel_java_tools/releases/javac11/v10.6/java_tools_javac11_linux-v10.6.zip
https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/b1c40e1de81913a3c40e5948f78719c28152486d.zip
https://mirror.bazel.build/github.com/bazelbuild/rules_java/archive/7cf3cefd652008d0a64a419c34c13bdca6c8f178.zip
https://mirror.bazel.build/openjdk/azul-zulu11.37.17-ca-jdk11.0.6/zulu11.37.17-ca-jdk11.0.6-linux_x64.tar.gz
SeanCurtis-TRI commented 3 years ago

Azul continues to haunt us. Maybe we should find the Agatekeepr and the Akeymaster.

jwnimmer-tri commented 3 years ago

The internet weather seems to have improved.

Also possibly https://github.com/bazelbuild/bazel/issues/12010 will be fixed as of Bazel 5.0, so we can close this until the storms come again.

jwnimmer-tri commented 2 years ago

We're seeing a notable uptick in JDK download errors:

Error downloading [https://mirror.bazel.build/openjdk/azul-zulu11.50.19-ca-jdk11.0.12/zulu11.50.19-ca-jdk11.0.12-linux_x64.tar.gz] to /media/ephemeral0/ubuntu/workspace/linux-focal-unprovisioned-gcc-bazel-continuous-snopt-packaging/_bazel_ubuntu/b761e54eae2c361f8c354b5d7ba3a90c/external/remotejdk11_linux/temp9555699724040762091/zulu11.50.19-ca-jdk11.0.12-linux_x64.tar.gz: Bytes read 61341232 but wanted 202599005
jwnimmer-tri commented 2 years ago

Setting --experimental_repository_downloader_retries to a non-zero value may help.

jwnimmer-tri commented 1 year ago

As of https://github.com/RobotLocomotion/drake-ci/pull/143, it seems we aren't seeing this anymore. Maybe internet weather is better, or maybe that CI patch or Bazel 5.0 fixed it. In any case, closing until it happens again.