FR: Support custom WWW-Authenticate schemes to extend known registry auth schemes.

[jbgcarnes]

the bzlmod pull extension doe not currently authenticate properly with remote repository. I believe the headers are set incorrectly as you will get a 401 Unauthorized when attempting to pull from a private registry.

example MODULE.bazel:

bazel_dep(name = "rules_oci", version = "0.3.8")
oci = use_extension("@rules_oci//oci:extensions.bzl", "oci")
oci.toolchains(
    name = "oci",
    crane_version = "v0.14.0",
    zot_version = "",
)

oci.pull(
   name = "private_distroless_static",
   image = "my.private.repo/distroless/static-debian11",
   digest = "sha256:sha_digest",
)

use_repo(oci, "private_distroless_static", "oci_crane_toolchains", "oci_crane_registry_toolchains")
register_toolchains("@oci_crane_toolchains//:all", "@oci_crane_registry_toolchains//:all")

in a BUILD.bazel file:

load("@rules_oci//oci:defs.bzl", "oci_image")

oci_image(
   name = "test",
   base = "@private_distroless_static",
...
)

This will produce the error:

rctx.download(
Error in download: java.io.IOException: Error in downloading my_registry/v2/distroless/static-debian11/manifest/sha256:sha
: GET returned 401 Unauthorized

I can confirm that the credentials are set:

echo "my.private.repo" | crane get auth
{"Username": "myuser", "Secret": "correct_secret"}

And adding a print in oci/pull.bzl:

print("""
current auth is:
""", pattern)

does show it found the correct username and password.

Finally, I was able to pull images using those credentials using crane copy, so I know the credentials are correct.

I even tried modifying the auth pattern to use bearer tokens:

pattern = {
"type": "pattern",
"pattern": "Bearer <password>",
"password": "password",
}

which fails. Is there something I'm missing?

[alexeagle]

I think this is already fixed at HEAD, @jbgcarnes do you mind syncing to the latest and see if you can reproduce?

[jtcarnes]

Will test out Monday and report back

[jbgcarnes]

@alexeagle tried it out this morning, it still returns 401 unauthorized (tain, tested locally with crane pull and crane will download)

I tested this with commit 629355

In MODULE.bazel

bazel_dep(name= "rules_oci", version = "0.3.9")

# Providing git override for testing HEAD
git_override(
  module_name = "rules_oci",
  # for brevity, simplified here
  remote = rules_oci_url,
  commit = "62935547208...."
)
# toolchain and image
oci_toolchain = use_extension("@rules_oci//oci:extensions.bzl", "oci")
oci_toolchain.toolchains(
  name = "oci",
  crane_version = "v0.14.0",
  zot_version = "",
)

oci_toolchain.pull(
  name = "distroless_static",
  image = "private-registry/distroless/static-debian11",
  digest = "sha256...."
)

use_repo(oci_toolchain, "distroless_static", "oci_crane_toolchains", "oci_crane_registry_toolchains")
register_toolchains("@oci_crane_toolchains//:all", "@oci_crane_registry_toolchains//:all")

This produces: java.io.IOException: error downloading .... : GET returned 401 Unauthorized

[jbgcarnes]

Could it be the changes that you did to oci_pull weren't then incorporated into the MODULES? (I can see that there was some code repetition in your TODO comment.

[thesayyn]

Could it be the changes that you did to oci_pull weren't then incorporated into the MODULES? (I can see that there was some code repetition in your TODO comment.

I see it now. thanks for the hint.

[alexeagle]

Yeah the syntax sugar change I just made in the oci_pull macro also needs to be made a second time in the module extension.

[thesayyn]

@alexeagle i am taking this too.

[elmarcelest]

I'm having the same problem without using bzlmod, is this expected? From the discussion above it sounded to me like using WORKSPACE & http_archive to pull the latest version should work, but I'm getting a 401 as well.

I'm using this in the workspace

http_archive(
    name = "rules_oci",
    sha256 = "076801d2128b072864241a3ed09f8557ff41737c6781b42579087c29686aefa7",
    strip_prefix = "rules_oci-80e9dd7d9c0deceed2d83e82bcf5c9c27d1bd90f",
    url = "https://github.com/bazel-contrib/rules_oci/archive/80e9dd7d9c0deceed2d83e82bcf5c9c27d1bd90f.tar.gz",
)

And then I have a private digital ocean registry

oci_pull(
    name = "chromium_nodejs_base",
    image = "registry.digitalocean.com/<whatever>/<whatever>",
    digest = "sha256:e061061a87ebff3ac23fb88b54ed3f18f1b3cb4a38b14271c53cede797de3cbe",
)

I am correctly logged in and the oci_push works but the oci_pull doesn't.

Thanks!!

[thesayyn]

I checked out the bzlmod extensions but couldn't find any issues over there. I'll need a reproduction repo for this. my guess is that private-registry and registry.digitalocean.com are not in the WWW-Authenticate scheme so we are not properly authenticating here.

[jbgcarnes]

To give you a solid target to try against: i'm using an internal gitlab container registry and getting the 401s (just tried it off of HEAD so still an issue)

[thesayyn]

taking this off the 1.0 milestone as we know this is not a bzlmod issue that needs to be fixed before 1.0. we can add support for additional WWW-Authenticate schemes later.

[nightrune]

This effectively blocks my org from using this. Is there any timeline then?

[thesayyn]

I need a repro to see what's broken.

[jhchabran]

@thesayyn I've reproduced the problem + poc'ed a fix over https://github.com/bazel-contrib/rules_oci/pull/236 if you want to take a look.

TL;DR, current code doesn't deal with "credsHelpers" which gcloud based auth uses.

[jbgcarnes]

There must be more then one issue then. My current issue does not use credsHelpers and still has this issue. My dockerconfig is more standard to something like gitlab, ie: ~/.docker/config.json

{
  "auths": {
      "my.gitlab.com": {
         "auth": "standard_token_base64"
      }
  }
}

[NilsIrl]

Facing {a similar,the same} issue. I see that the rules_oci maintains a hard-coded list of known registries that use WWW-authentication: https://github.com/avdv/rules_oci/blob/88cb9959caa86321b030314135bf935033de9cb4/oci/private/pull.bzl#L36

Could it be that the issue is happening on self-managed GitLab instances?

[jbgcarnes]

@NilsIrl you are 100% correct.

I am trying to pull and push images from a self managed Gitlab instance. The push works, as it parses the credential file at ~/.docker/config.json.

The problem is pulling the base image. That does not work as the rules currently only look for the hard-coded list of registries to use WWW-authentication.