Closed wskinner-dd closed 12 months ago
I was able to get past this. I believe the root cause is a credential caching issue within `rules_oci, but I haven't been able to reproduce the exact state that led to the issue.
What I did to resolve it:
docker logout
rm /root/.cache/bazel/_bazel_root/c6f23abe7349c03cc452f0032deaf22d/external/oci_auth_config/config.json
docker login
bazel build ...
I don't know how exactly my repository got into that state.
credential caching issue within `rules_oci
we don't have a caching mechanism that persists tokens across different bazel calls. if docker pull
works, then oci_pull should too
Ran into the same trouble:
oci_pull(
name = "nginx_alpine",
image = "docker.io/nginxinc/nginx-unprivileged",
# tag = "stable-alpine3.18",
digest = "sha256:25eb22bc64d5292b197c7de8c3b7022100ab3f4882054158f02560341d0085fa",
platforms = [
"linux/amd64",
"linux/arm64"
],
)
% bazel fetch @nginx_alpine
Starting local Bazel server and connecting to it...
INFO: Repository nginx_alpine_linux_amd64 instantiated at:
/Users/farcop/Sources/kublr/WORKSPACE:413:13: in <toplevel>
/Users/farcop/Sources/kublr/images.bzl:36:13: in fetch_images
/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/pull.bzl:121:22: in oci_pull
Repository rule oci_pull defined at:
/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl:437:27: in <toplevel>
INFO: repository @nginx_alpine_linux_amd64' used the following cache hits instead of downloading the corresponding file.
* Hash '25eb22bc64d5292b197c7de8c3b7022100ab3f4882054158f02560341d0085fa' for https://index.docker.io/v2/nginxinc/nginx-unprivileged/manifests/sha256:25eb22bc64d5292b197c7de8c3b7022100ab3f4882054158f02560341d0085fa
* Hash '67e31c34bd90e4d6fc2cf5c1ede699e50372e20bf2f6966c4ae2bb4c9ae3682b' for https://index.docker.io/v2/nginxinc/nginx-unprivileged/manifests/sha256:67e31c34bd90e4d6fc2cf5c1ede699e50372e20bf2f6966c4ae2bb4c9ae3682b
If the definition of 'repository @nginx_alpine_linux_amd64' was updated, verify that the hashes were also updated.
WARNING: Download from https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696 failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 401 Unauthorized
ERROR: An error occurred during the fetch of repository 'nginx_alpine_linux_amd64':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 383, column 29, in _oci_pull_impl
downloader.download_blob(image_mf["config"]["digest"], image_config_file)
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 281, column 61, in lambda
download_blob = lambda identifier, output: _download(rctx, state, identifier, output, "blobs"),
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 225, column 23, in _download
return download_fn(
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/download.bzl", line 130, column 25, in _bazel_download
return rctx.download(**kwargs)
Error in download: java.io.IOException: Error downloading [https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696] to /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/nginx_alpine_linux_amd64/63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696: GET returned 401 Unauthorized
ERROR: /Users/farcop/Sources/kublr/WORKSPACE:413:13: fetching oci_pull rule //external:nginx_alpine_linux_amd64: Traceback (most recent call last):
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 383, column 29, in _oci_pull_impl
downloader.download_blob(image_mf["config"]["digest"], image_config_file)
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 281, column 61, in lambda
download_blob = lambda identifier, output: _download(rctx, state, identifier, output, "blobs"),
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 225, column 23, in _download
return download_fn(
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/download.bzl", line 130, column 25, in _bazel_download
return rctx.download(**kwargs)
Error in download: java.io.IOException: Error downloading [https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696] to /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/nginx_alpine_linux_amd64/63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696: GET returned 401 Unauthorized
ERROR: Evaluation of query "deps(@nginx_alpine)" failed: preloading transitive closure failed: no such package '@nginx_alpine_linux_amd64//': java.io.IOException: Error downloading [https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696] to /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/nginx_alpine_linux_amd64/63c8b878873689a9cb38a0e35a9c99f9b505a024a53bda00d1c04c7a4696e696: GET returned 401 Unauthorized
Loading: 2 packages loaded
Fetching repository @nginx_alpine_linux_arm64; starting
I tried bazel clean and delete repository_cache. Nothing helps...
docker pull works perfectly, but oci_pull - not
@farcop Interesting. I'm curious what ls -l /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/oci_auth_config/config.json
shows?
@wskinner-dd wow! its symlink on the valid config.json I suppose
lrwxr-xr-x 1 farcop wheel 33 сен 16 14:17 /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/oci_auth_config/config.json -> /Users/farcop/.docker/config.json
cat /Users/farcop/.docker/config.json
{
"auths": {
"https://index.docker.io/v2/": {
"auth": "Zm...=="
},
"https://index.docker.io/v1/": {
"auth": "Zm...="
}
}
}
@farcop I suspect if you remove that symlink as well as its target, then reauthenticate to your docker registry, you'll get past the issue.
% rm /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/oci_auth_config/config.json
% rm /Users/farcop/.docker/config.json
% docker login
Authenticating with existing credentials...
Login Succeeded
% bazel fetch @node_alpine
Starting local Bazel server and connecting to it...
INFO: Repository node_alpine instantiated at:
/Users/farcop/Sources/kublr/WORKSPACE:413:13: in <toplevel>
/Users/farcop/Sources/kublr/images.bzl:84:13: in fetch_images
/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/pull.bzl:143:14: in oci_pull
Repository rule oci_alias defined at:
/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl:529:28: in <toplevel>
ERROR: An error occurred during the fetch of repository 'node_alpine':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 477, column 36, in _oci_alias_impl
downloader = _create_downloader(rctx)
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 276, column 40, in _create_downloader
"config": json.decode(rctx.read(rctx.attr.config)),
Error in read: Not a regular file: /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/oci_auth_config/config.json
ERROR: /Users/farcop/Sources/kublr/WORKSPACE:413:13: fetching oci_alias rule //external:node_alpine: Traceback (most recent call last):
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 477, column 36, in _oci_alias_impl
downloader = _create_downloader(rctx)
File "/private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/rules_oci/oci/private/pull.bzl", line 276, column 40, in _create_downloader
"config": json.decode(rctx.read(rctx.attr.config)),
Error in read: Not a regular file: /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/oci_auth_config/config.json
ERROR: no such package '@node_alpine//': Not a regular file: /private/var/tmp/_bazel_farcop/e66529431dbede3bc951f821053136ab/external/oci_auth_config/config.json
Loading: 0 packages loaded
could you remove the config file from ~/.docker/config.json
and run docker login again?
After all bazel cashes cleaned
$ bazel fetch @nginx_alpine
WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
WARNING: Could not fetch the manifest. Either there was an authentication issue or trying to pull an image with OCI image media types.
Falling back to using `curl`. See https://github.com/bazelbuild/bazel/issues/17829 for the context.
INFO: Repository nginx_alpine_linux_amd64 instantiated at:
/home/kublr/kublr/WORKSPACE:413:13: in <toplevel>
/home/kublr/kublr/images.bzl:36:13: in fetch_images
/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/pull.bzl:121:22: in oci_pull
Repository rule oci_pull defined at:
/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl:437:27: in <toplevel>
WARNING: Download from https://index.docker.io/v2/nginxinc/nginx-unprivileged/manifests/sha256:bb1a470f475dfe3d11ccfc2ac53177200d3ba81e9e907cf4ed0185759a46e583 failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 401 Unauthorized
WARNING: Download from https://index.docker.io/v2/nginxinc/nginx-unprivileged/manifests/sha256:5bb412ff61fef36458618c6ab4d08dcb36bb85356dfd03a9fb36af8b6e6b92a3 failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 401 Unauthorized
WARNING: Download from https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a failed: class com.google.devtools.build.lib.bazel.repository.downloader.UnrecoverableHttpException GET returned 401 Unauthorized
INFO: Repository nginx_alpine_linux_arm64 instantiated at:
/home/kublr/kublr/WORKSPACE:413:13: in <toplevel>
/home/kublr/kublr/images.bzl:36:13: in fetch_images
/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/pull.bzl:121:22: in oci_pull
Repository rule oci_pull defined at:
/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl:437:27: in <toplevel>
ERROR: An error occurred during the fetch of repository 'nginx_alpine_linux_amd64':
Traceback (most recent call last):
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl", line 383, column 29, in _oci_pull_impl
downloader.download_blob(image_mf["config"]["digest"], image_config_file)
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl", line 281, column 61, in lambda
download_blob = lambda identifier, output: _download(rctx, state, identifier, output, "blobs"),
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl", line 225, column 23, in _download
return download_fn(
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/download.bzl", line 130, column 25, in _bazel_download
return rctx.download(**kwargs)
Error in download: java.io.IOException: Error downloading [https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a] to /home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/nginx_alpine_linux_amd64/06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a: GET returned 401 Unauthorized
ERROR: /home/kublr/kublr/WORKSPACE:413:13: fetching oci_pull rule //external:nginx_alpine_linux_amd64: Traceback (most recent call last):
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl", line 383, column 29, in _oci_pull_impl
downloader.download_blob(image_mf["config"]["digest"], image_config_file)
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl", line 281, column 61, in lambda
download_blob = lambda identifier, output: _download(rctx, state, identifier, output, "blobs"),
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/pull.bzl", line 225, column 23, in _download
return download_fn(
File "/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/rules_oci/oci/private/download.bzl", line 130, column 25, in _bazel_download
return rctx.download(**kwargs)
Error in download: java.io.IOException: Error downloading [https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a] to /home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/nginx_alpine_linux_amd64/06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a: GET returned 401 Unauthorized
INFO: repository @nginx_alpine_linux_arm64' used the following cache hits instead of downloading the corresponding file.
* Hash 'bb1a470f475dfe3d11ccfc2ac53177200d3ba81e9e907cf4ed0185759a46e583' for file:/home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/nginx_alpine_linux_arm64/.output/manifest.json
If the definition of 'repository @nginx_alpine_linux_arm64' was updated, verify that the hashes were also updated.
ERROR: Evaluation of query "deps(@nginx_alpine)" failed: preloading transitive closure failed: no such package '@nginx_alpine_linux_amd64//': java.io.IOException: Error downloading [https://index.docker.io/v2/nginxinc/nginx-unprivileged/blobs/sha256:06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a] to /home/kublr/.cache/bazel/_bazel_kublr/69fb4c52c15303cc4e23e5352527edec/external/nginx_alpine_linux_amd64/06f0f53713bb14a22bc396231b8f89d0b05e1d3c84b3766014895b1a245c3b7a: GET returned 401 Unauthorized
Loading: 1 packages loaded
Note that oci_pull from other public registry works well.
@thesayyn Finally, I found the reason! Just recently I added to the project rules_apko and done this https://github.com/chainguard-dev/rules_apko/blob/main/docs/initial-setup.md As soon as I rolled back these changes everything worked.
I think we should make a bug in rules_apko
Going to close this as this was a misusage. #361 should prevent this in the future.
I'm working on migrating my company to
rules_oci
.rules_oci
version:v1.3.1
bazel
version:6.3.2
I add a custom base image like this:
In my
BUILD
file, I useoci_image
to depend on that base image:When building on my linux host, this works as expected. But when building within a Docker container,
oci_pull
produces the following error:I have verified the following:
docker pull
the image from within the container:/root/.docker/config.json
in the container.