search

monogon-dev / monogon

The Monogon Monorepo. May contain traces of peanuts and a ✨pure Go Linux userland✨. Work in progress!
https://monogon.tech
Apache License 2.0
377 stars 9 forks source link

Unable to bootstrap cluster due to missing artifacts. #297

Closed josephglanville closed 6 months ago

josephglanville commented 6 months ago

Hi, I'm trying to get started with Monogon but currently can't bootstrap due to missing artifacts, see attached Bazel build log:

$ bazel run //metropolis:launch-cluster
IntelliJ found at /home/jpg/.local/share/JetBrains/IntelliJIdea2023.3, aspect repository already patched.
INFO: Invocation ID: cef4f201-628e-427f-a55d-c55210ac6127
WARNING: Download from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Repository linux-firmware instantiated at:
  /home/jpg/src/monogon/WORKSPACE:183:24: in <toplevel>
  /home/jpg/src/monogon/third_party/linux-firmware/external.bzl:21:17: in linux_firmware_external
Repository rule http_archive defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'linux-firmware':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/WORKSPACE:183:24: fetching http_archive rule //external:linux-firmware: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:33:22: //metropolis/node:firmware depends on @@linux-firmware//:all_files in repository @@linux-firmware which failed to fetch. no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:33:22: //metropolis/node:firmware depends on @@linux-firmware//:metadata in repository @@linux-firmware which failed to fetch. no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:40:11: //metropolis/node:ucode depends on @@linux-firmware//:amd_ucode in repository @@linux-firmware which failed to fetch. no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp790376834398890193/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
INFO: Repository rules_oci~1.6.0~oci~distroless_base instantiated at:
  <builtin>: in <toplevel>
Repository rule oci_alias defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl:500:28: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_oci~1.6.0~oci~distroless_base':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 449, column 55, in _oci_alias_impl
        manifest, _, digest = downloader.download_manifest(rctx.attr.identifier, "mf.json")
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 322, column 74, in lambda
        download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 265, column 23, in _download_manifest
        result = _download(rctx, state, identifier, output, "manifests", allow_fail = True)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 230, column 22, in _download
        auth = _get_token(rctx, state, rctx.attr.registry, rctx.attr.repository)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 137, column 24, in _get_token
        pattern = _get_auth(rctx, state, registry)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 98, column 55, in _get_auth
        pattern = _fetch_auth_via_creds_helper(rctx, host_raw, helper_val)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 189, column 13, in _fetch_auth_via_creds_helper
        fail("credential helper failed: \nSTDOUT:\n{}\nSTDERR:\n{}".format(result.stdout, result.stderr))
Error in fail: credential helper failed:
STDOUT:

STDERR:
/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0~oci~distroless_base/gcloud.sh: line 2: exec: docker-credential-gcloud: not found
ERROR: <builtin>: fetching oci_alias rule //:rules_oci~1.6.0~oci~distroless_base: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 449, column 55, in _oci_alias_impl
        manifest, _, digest = downloader.download_manifest(rctx.attr.identifier, "mf.json")
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 322, column 74, in lambda
        download_manifest = lambda identifier, output: _download_manifest(rctx, state, identifier, output),
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 265, column 23, in _download_manifest
        result = _download(rctx, state, identifier, output, "manifests", allow_fail = True)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 230, column 22, in _download
        auth = _get_token(rctx, state, rctx.attr.registry, rctx.attr.repository)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 137, column 24, in _get_token
        pattern = _get_auth(rctx, state, registry)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 98, column 55, in _get_auth
        pattern = _fetch_auth_via_creds_helper(rctx, host_raw, helper_val)
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0/oci/private/pull.bzl", line 189, column 13, in _fetch_auth_via_creds_helper
        fail("credential helper failed: \nSTDOUT:\n{}\nSTDERR:\n{}".format(result.stdout, result.stderr))
Error in fail: credential helper failed:
STDOUT:

STDERR:
/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0~oci~distroless_base/gcloud.sh: line 2: exec: docker-credential-gcloud: not found
ERROR: no such package '@@rules_oci~1.6.0~oci~distroless_base//': credential helper failed:
STDOUT:

STDERR:
/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0~oci~distroless_base/gcloud.sh: line 2: exec: docker-credential-gcloud: not found
ERROR: /home/jpg/src/monogon/metropolis/test/e2e/preseedtest/BUILD.bazel:36:10: //metropolis/test/e2e/preseedtest:preseedtest_image depends on @@rules_oci~1.6.0~oci~distroless_base//:distroless_base in repository @@rules_oci~1.6.0~oci~distroless_base which failed to fetch. no such package '@@rules_oci~1.6.0~oci~distroless_base//': credential helper failed:
STDOUT:

STDERR:
/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/rules_oci~1.6.0~oci~distroless_base/gcloud.sh: line 2: exec: docker-credential-gcloud: not found
WARNING: Download from https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Repository xfsprogs instantiated at:
  /home/jpg/src/monogon/WORKSPACE:215:18: in <toplevel>
  /home/jpg/src/monogon/third_party/xfsprogs/external.bzl:25:17: in xfsprogs_external
Repository rule http_archive defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'xfsprogs':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp6002759322421218556/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/WORKSPACE:215:18: fetching http_archive rule //external:xfsprogs: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp6002759322421218556/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@xfsprogs//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp6002759322421218556/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
WARNING: Download from https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Repository cap instantiated at:
  /home/jpg/src/monogon/WORKSPACE:272:13: in <toplevel>
  /home/jpg/src/monogon/third_party/cap/external.bzl:24:17: in cap_external
Repository rule http_archive defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'cap':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp14842024153914983823/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/WORKSPACE:272:13: fetching http_archive rule //external:cap: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp14842024153914983823/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@cap//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp14842024153914983823/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/chrony/BUILD.bazel:138:11: @@chrony//:sys depends on @@cap//:cap in repository @@cap which failed to fetch. no such package '@@cap//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp14842024153914983823/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:49:12: //metropolis/node:rootfs depends on @@xfsprogs//:mkfs in repository @@xfsprogs which failed to fetch. no such package '@@xfsprogs//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp6002759322421218556/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
INFO: Analyzed target //metropolis/test/launch/cli/launch-cluster:launch-cluster (0 packages loaded, 0 targets configured).
WARNING: errors encountered while analyzing target '//metropolis/test/launch/cli/launch-cluster:launch-cluster', it will not be built.
Analysis failed
INFO: Found 0 targets...
ERROR: command succeeded, but there were loading phase errors
INFO: Elapsed time: 0.364s, Critical Path: 0.05s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
josephglanville commented 6 months ago

So trying to debug one part of that I double checked that docker-credential-gcloud was on PATH and it is:

$ which docker-credential-gcloud
/home/jpg/.google-cloud-sdk/bin/docker-credential-gcloud

Which is strange because I actually use GAR with Bazel and rules_oci for another project and it works fine.

josephglanville commented 6 months ago

Artifacts from kernel.org do indeed appear to be missing though:

$ wget https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz
--2024-03-29 23:44:20--  https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving git.kernel.org (git.kernel.org)... 2604:1380:40e1:4800::1, 145.40.73.55
Connecting to git.kernel.org (git.kernel.org)|2604:1380:40e1:4800::1|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2024-03-29 23:44:24 ERROR 404: Not Found.
josephglanville commented 6 months ago

Ok so oci.pull mystery solved. My Docker config is setup to use helper to download from gcr.io because I pull and push private images there, for some reason this doesn't work with this Bazel setup but does with mine (likely different sandboxing settings?). I was able to work around that part by invoking Bazel with DOCKER_CONFIG="", i.e disable my own configuration and use Docker defaults.

josephglanville commented 6 months ago

This gets it down to just these missing/not available URLs which I am not sure are truly unavailable or just not available to me given I am based in Thailand and country specific blocking is unfortunately something I run into quite often:

$ DOCKER_CONFIG="" bazel run //metropolis:launch-cluster
IntelliJ found at /home/jpg/.local/share/JetBrains/IntelliJIdea2023.3, aspect repository already patched.
INFO: Invocation ID: ee54593d-325a-4112-ba33-06cd236f2fdd
WARNING: Download from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Repository linux-firmware instantiated at:
  /home/jpg/src/monogon/WORKSPACE:183:24: in <toplevel>
  /home/jpg/src/monogon/third_party/linux-firmware/external.bzl:21:17: in linux_firmware_external
Repository rule http_archive defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'linux-firmware':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/WORKSPACE:183:24: fetching http_archive rule //external:linux-firmware: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:33:22: //metropolis/node:firmware depends on @@linux-firmware//:all_files in repository @@linux-firmware which failed to fetch. no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:33:22: //metropolis/node:firmware depends on @@linux-firmware//:metadata in repository @@linux-firmware which failed to fetch. no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:40:11: //metropolis/node:ucode depends on @@linux-firmware//:amd_ucode in repository @@linux-firmware which failed to fetch. no such package '@@linux-firmware//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/linux-firmware/temp906252692376987136/linux-firmware-20231211.tar.gz: GET returned 404 Not Found
WARNING: Could not find the `$HOME/.docker/config.json` and `$XDG_RUNTIME_DIR/containers/auth.json` file.
 Running one of `podman login`, `docker login`, `crane login` may help.
WARNING: Download from https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Repository cap instantiated at:
  /home/jpg/src/monogon/WORKSPACE:272:13: in <toplevel>
  /home/jpg/src/monogon/third_party/cap/external.bzl:24:17: in cap_external
Repository rule http_archive defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'cap':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp3423818194129872951/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/WORKSPACE:272:13: fetching http_archive rule //external:cap: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp3423818194129872951/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@cap//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp3423818194129872951/v1.2.55.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/chrony/BUILD.bazel:138:11: @@chrony//:sys depends on @@cap//:cap in repository @@cap which failed to fetch. no such package '@@cap//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/libs/libcap/libcap.git/snapshot/libcap-cap/v1.2.55.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/cap/temp3423818194129872951/v1.2.55.tar.gz: GET returned 404 Not Found
WARNING: Download from https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Repository xfsprogs instantiated at:
  /home/jpg/src/monogon/WORKSPACE:215:18: in <toplevel>
  /home/jpg/src/monogon/third_party/xfsprogs/external.bzl:25:17: in xfsprogs_external
Repository rule http_archive defined at:
  /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl:384:31: in <toplevel>
ERROR: An error occurred during the fetch of repository 'xfsprogs':
   Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp8836237225728853799/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/WORKSPACE:215:18: fetching http_archive rule //external:xfsprogs: Traceback (most recent call last):
    File "/home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/bazel_tools/tools/build_defs/repo/http.bzl", line 145, column 45, in _http_archive_impl
        download_info = ctx.download_and_extract(
Error in download_and_extract: java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp8836237225728853799/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
ERROR: no such package '@@xfsprogs//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp8836237225728853799/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
ERROR: /home/jpg/src/monogon/metropolis/node/BUILD.bazel:49:12: //metropolis/node:rootfs depends on @@xfsprogs//:mkfs in repository @@xfsprogs which failed to fetch. no such package '@@xfsprogs//': java.io.IOException: Error downloading [https://git.kernel.org/pub/scm/fs/xfs/xfsprogs-dev.git/snapshot/xfsprogs-dev-5.10.0.tar.gz] to /home/jpg/.cache/bazel/_bazel_jpg/d7f18bc350c43d8020bc6f28b99ca7e9/external/xfsprogs/temp8836237225728853799/xfsprogs-dev-5.10.0.tar.gz: GET returned 404 Not Found
INFO: Analyzed target //metropolis/test/launch/cli/launch-cluster:launch-cluster (3 packages loaded, 22 targets configured).
WARNING: errors encountered while analyzing target '//metropolis/test/launch/cli/launch-cluster:launch-cluster', it will not be built.
Analysis failed
INFO: Found 0 targets...
ERROR: command succeeded, but there were loading phase errors
INFO: Elapsed time: 69.429s, Critical Path: 0.06s
INFO: 1 process: 1 internal.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target
leoluk commented 6 months ago

Thank you for the bug report and the troubleshooting!

Hmm - there's definitely something weird going on. The URLs do work for me (I tried https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz).

I would be surprised if git.kernel.org did geoblocking, but perhaps something's wrong with the CDN node closest to you. We'll try to reproduce the issue.

This bug report is an excellent example for why we need https://github.com/monogon-dev/monogon/issues/178 before our first stable release :-)

josephglanville commented 6 months ago

Yeah it's a geo-block or mirror problem. I connected to a VPN node in Belgium and was able to download the archive.

leoluk commented 6 months ago

@josephglanville Could you please try again whether you can reach https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/snapshot/linux-firmware-20231211.tar.gz now?

We're no longer able to reproduce it, perhaps the CDN mirror has been fixed.

josephglanville commented 6 months ago

Yes it's fixed now!

NODE ID                                       STATE   ADDRESS    HEALTH    ROLES                                  TPM   VERSION                   HEARTBEAT
metropolis-f78f41dfe71f307e91a8eceadc5e6e61   UP      10.1.0.2   HEALTHY   ConsensusMember,KubernetesController   yes   v0.1.0-dev561.gae076614   2s
metropolis-fa1d0758437de8f0ef96cbb06dad6b0f   UP      10.1.0.4   HEALTHY   KubernetesWorker                       yes   v0.1.0-dev561.gae076614   4s
metropolis-fc0500b89d26283335e6f6bbc5a7ef32   UP      10.1.0.3   HEALTHY   KubernetesWorker                       yes   v0.1.0-dev561.gae076614   4s
leoluk commented 6 months ago

Yay :) Thank you for your bug report!