[Bazel CI] [ IOException]no such package '@@glib//' error with Bazel@HEAD

[sgowroji]

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3856#018f7548-3f1d-4293-8931-28b75d5d2372

Platform: Ubuntu

Logs:

ERROR: no such package '@@glib//': java.io.IOException: Error downloading [https://download.gnome.org/sources/glib/2.75/glib-2.75.0.tar.xz] to /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f95649d44a2292dba9779ac70640b5e9/external/glib/temp13126141629506343744/glib-2.75.0.tar.xz: GET returned 503 No healthy backends
ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f95649d44a2292dba9779ac70640b5e9/external/rules_foreign_cc_examples_third_party/glib/BUILD:10:11: @@rules_foreign_cc_examples_third_party//glib:glib_build_test_0__deps depends on @@glib//:glib in repository @@glib which failed to fetch. no such package '@@glib//': java.io.IOException: Error downloading [https://download.gnome.org/sources/glib/2.75/glib-2.75.0.tar.xz] to /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/f95649d44a2292dba9779ac70640b5e9/external/glib/temp13126141629506343744/glib-2.75.0.tar.xz: GET returned 503 No healthy backends
ERROR: Analysis of target '@@rules_foreign_cc_examples_third_party//glib:glib_build_test' failed; 

Steps:

git clone https://github.com/bazelbuild/rules_foreign_cc
git reset 7ce62009557d73da9aa0d2a1ca7eded49078b3cf  --hard
export USE_BAZEL_VERSION=bf2f762f815cc5fd0a4310c66f1633e415177521 

CC Greenteam @fweikert

[jsharpe]

This is a flake due to an outage of the upstream gnome download servers - the URL seems to be happy now so I suspect this will go away on the next CI run..

[sgowroji]

@jsharpe We are still seeing this error in downstream https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/3900#018fe16d-3dc9-4e18-b528-30382365b2b5

Cc @Wyverald @meteorcloudy

[jsharpe]

This seems to be an infra flake on gitlab.freedesktop.org this time which is different to the original issue. I guess this could be made more resilient if the packages in question are put on the bazel gcs mirror? @sgowroji do you have the bandwidth available to put those issue requests in on bazelbuild/bazel and place a PR on this repo to add the mirrored urls?

[sgowroji]

@jsharpe Could you please file an issue under mirror request. I can update there. Thanks for looking into it.

[jsharpe]

@sgowroji I know how to request it - I'm asking if you have bandwidth to do this as I don't currently have time to do this for all the URLs that need mirroring that aren't yet in this repo as I know there are a few, its not just the one that is listed in this issue.

[sgowroji]

Sorry am not sure about the list of URL's. CC @meteorcloudy Could you please guide on this mirror links.

[meteorcloudy]

@jsharpe This is beyond the responsibility of our gTech operational team. If you can get the list of URLs to mirror, they can help that. But the team doesn't have the expertise or bandwidth to dig into a specific project.

[jsharpe]

@meteorcloudy the effort is in extracting the list of URLs to mirror, and then adding in the mirror configuration to this repo. I have no knowledge of the internal structure of the Google teams, nor their remit, is there any public documentation around this commitment and levels we should expect?

The reality is this was a infrastructure flake from a third party server that caused an intermittent failure in bazel CI; really bazel CI should have a download cache / pull through proxy, not only to mitigate against such flakes but also to protect it against a large number of downloads / high bandwidth utilisation. It's also my understanding that at some point mirror.bazel.build will go away at which point the download cache would become more important and solve this issue?

The bazel at head plus downstream is primarily an early warning of breaks in future bazel versions and given the very limited resource available to maintain this repo (my time is entirely voluntary and on my own time), I just don't have any bandwidth to address such issues, particularly as there seems to have been a higher than average number of breaking changes in the bazel 8 series, which in all cases for this repo have just been down to transitive dependencies of the examples which can take a not insignificant amount of time to track down, especially if you are unaware of the context of the changes in bazel HEAD (e.g. the aar_import failure in #1203).

[jsharpe]

I have opened #1214 but I have no intention of spending anytime fixing this unless someone wants to sponsor me to do so.