Struggling to understand how to configure containerd.toml for a private registry

[DazWilkin]

I found this and #835 but I receive unauthorized errors when trying to pull an image from GCR:

level=error msg="PullImage "gcr.io/[redacted]/hello-world:latest" failed" error="failed to resolve image "gcr.io/[redacted]/hello-world:latest": no available registry endpoint: unexpected status code https://gcr.io/v2/[redacted]/hello-world/manifests/latest: 401 Unauthorized"

I'm using MicroK8s with Krustlet and the Krustlet does not yet support imagePullSecrets so I'd like to configure MicroK8s' containerd to authenticate.

Per the docs, I'm replicating the approach used with Docker:

gcloud auth print-access-token \
| docker login \
-u oauth2accesstoken \
--password-stdin \
https://gcr.io

more ${HOME}/.docker/config.json
{
    "auths": {
        "https://gcr.io": {
            "auth": "b2F1dGgy..."
        }
    }
}

I appreciate that I'll have limited time to use the token but...

I tried:

...
[plugins]
  ...
  [plugins.cri]
    ...
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://localhost:32000"]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = ["https://gcr.io"]
      [plugins.cri.registry.configs]
        [plugins.cri.registry.configs.auths]
          [plugins.cri.registry.configs.auths."https://gcr.io"]
            auth = "b2F1dGgy..."
...

I'm restarting MicroK8s after changes.

Using Docker against the registry, works:

docker image ls gcr.io/[redacted]/hello-world
REPOSITORY          TAG                 IMAGE ID            CREATED

docker pull gcr.io/[redacted]/hello-world
Using default tag: latest
latest: Pulling from [redacted]/hello-world
Digest: sha256:90659bf80b44ce6be8234e6ff90a1ac34acbeb826903b02cfa0da11c82cbc042
Status: Downloaded newer image for gcr.io/[redacted]/hello-world:latest

docker image ls gcr.io/[redacted]/hello-world
REPOSITORY                      TAG                 IMAGE ID            CREATED
gcr.io/[redacted]/hello-world   latest              bf756fb1ae65        4 months ago

Any help would be appreciated.

[DazWilkin]

Thank you.

The node's kubelet (Krustlet) does not (yet) implement using imagePullSecrets.

Am I unable to do this by some form of the configuration I show?

[mikebrow]

hmm :)

[mikebrow]

let's see the whole config

[DazWilkin]

It's mostly the default configuration created by MicroK8s:

oom_score = 0

[grpc]
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[debug]
  address = ""
  uid = 0
  gid = 0

[metrics]
  address = "127.0.0.1:1338"
  grpc_histogram = false

[cgroup]
  path = ""

[plugins]
  [plugins.cgroups]
    no_prometheus = false
  [plugins.cri]
    stream_server_address = "127.0.0.1"
    stream_server_port = "0"
    enable_selinux = false
    sandbox_image = "k8s.gcr.io/pause:3.1"
    stats_collect_period = 10
    systemd_cgroup = false
    enable_tls_streaming = false
    max_container_log_line_size = 16384
    [plugins.cri.containerd]
      snapshotter = "overlayfs"
      no_pivot = false
      [plugins.cri.containerd.default_runtime]
        runtime_type = "io.containerd.runtime.v1.linux"
        runtime_engine = ""
        runtime_root = ""
      [plugins.cri.containerd.untrusted_workload_runtime]
        runtime_type = ""
        runtime_engine = ""
        runtime_root = ""
    [plugins.cri.cni]
      bin_dir = "${SNAP}/opt/cni/bin"
      conf_dir = "${SNAP_DATA}/args/cni-network"
      conf_template = ""
    [plugins.cri.registry]
      [plugins.cri.registry.mirrors]
        [plugins.cri.registry.mirrors."docker.io"]
          endpoint = ["https://registry-1.docker.io"]
        [plugins.cri.registry.mirrors."local.insecure-registry.io"]
          endpoint = ["http://localhost:32000"]
        [plugins.cri.registry.mirrors."gcr.io"]
          endpoint = ["https://gcr.io"]
      [plugins.cri.registry.configs]
        [plugins.cri.registry.configs.auths]
          [plugins.cri.registry.configs.auths."https://gcr.io"]
            auth = "b2F1dGgy..."
  [plugins.diff-service]
    default = ["walking"]
  [plugins.linux]
    shim = "containerd-shim"
    runtime = "${RUNTIME}"
    runtime_root = ""
    no_shim = false
    shim_debug = true
  [plugins.scheduler]
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = "0s"
    startup_delay = "100ms"

NOTE I've redacted the token value.

NOTE I was unsure whether the hierarchy needed to be explicitly defined e.g. [plugins.cri.registry.configs] and [plugins.cri.registry.configs.auth]

In the comment reply that I referenced, a different value was used [plugins.cri.registry.auths] (no configs)

[mikebrow]

[plugins.cri.registry.auths] [plugins.cri.registry.auths."https://gcr.io"] username = "" password = "" auth = "" identitytoken = ""

[mikebrow]

        // Username is the username to login the registry.
Username string `toml:"username" json:"username"`
// Password is the password to login the registry.
Password string `toml:"password" json:"password"`
// Auth is a base64 encoded string from the concatenation of the username,
// a colon, and the password.
Auth string `toml:"auth" json:"auth"`
// IdentityToken is used to authenticate the user and get
// an access token for the registry.
IdentityToken string `toml:"identitytoken" json:"identitytoken"

[mikebrow]

seems to be missing v2

[DazWilkin]

Hmmm... this suggests that the auth value from e.g. ${HOME}/.docker/config.json. is not the same as this auth value. I would, e.g.:

gcloud auth print-access-token \
| docker login -u oauth2accesstoken --password-stdin https://gcr.io

This updates and entry in ${HOME}/.docker/config.json e.g.

{
    "auths": {
        "https://gcr.io": {
            "auth": "b2F1dGgy..."
        }
    }
}

And I've been mirroring the same value in the containerd TOML.

Yes, I am confused by the spec but it suggests that I don't need to explicit state the v2!?

I will look into it some more.

Thank you!

[DazWilkin]

I'm going to take my dog for a swim...

Will look at this tomorrow.

Thanks for looking into it for me!

[DazWilkin]

I've tried most permutations of username|password|auth and I'm unsuccessful :-(

[mikebrow]

Try this: https://github.com/ubuntu/microk8s/issues/990#issuecomment-592296296

[mikebrow]

if that works out for you I'll fix the docs ..

[DazWilkin]

It does not :-(

But, I mostly understand why that approach should work. It seems redundant to have to provide the username and password twice; once directly and then a second time base64 encoded.

I'd prefer to not use a long-lived token (from a service account) as is done in that example (see Authentication Methods).

The configuration is confusing. In this (and others) examples, the registry auth is referenced as:

registry.auths

But, in the sources:

https://github.com/containerd/cri/blob/8252e54f936b85b58799600edcb98987a8665300/pkg/config/config.go#L181 https://github.com/containerd/cri/blob/8252e54f936b85b58799600edcb98987a8665300/pkg/config/config.go#L144 https://github.com/containerd/cri/blob/8252e54f936b85b58799600edcb98987a8665300/pkg/config/config.go#L155

I've tried registry.auths and because registry.configs is a map[string]X that mirrors (sic.) mirrors, I tried the following too:

[plugins.cri.registry.configs."https://gcr.io".auth]

None of these work.

Is there some way that I can inspect the URL that it is try to make against the registry?

journalctl logs:

failed to resolve image "gcr.io/[redacted]/hello-world:latest": no available registry endpoint: unexpected status code https://gcr.io/v2/[redacted]/hello-world/manifests/latest: 401 Unauthorized"

But, if I take $(gcloud auth print-access-token) (which is what I'm using for the auth value), I can:

curl \
--request GET\
 --header "Authorization: Bearer $(gcloud auth print-access-token)" \
https://gcr.io/v2/[redacted]/hello-world/manifests/latest
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1510,
      "digest": "sha256:bf756fb1..."
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 2529,
         "digest": "sha256:0e03bdcc..."
      }
   ]
}

NOTE this is the same URL as returns 401 from containerd

[mikebrow]

restarting micro k8s... is that the parent for the instance of containerd? if not you need to restart containerd..

[DazWilkin]

IIUC, yes.

microk8s.stop && microk8s.start

yields:

stop of [... microk8s.daemon-containerd ...]

[mikebrow]

it may help to run containerd in debug (-D) but pretty sure we don't output the auth details for privacy reasons..

[DazWilkin]

IIUC dockerd leverages containerd and I'm able to find 2 containerd processes running (one for Docker and the other for MicroK8s). Unfortunately, while the MicroK8s configuration uses containerd.toml to define these registries and auths, I'm unable to correlate this to the containerd instance run by Docker which works with the GCR registry; if I docker login, that config is stored in config.json and doesn't appear to be reflected explicity in the containerd.toml.

Perhaps I should ping the comment you cited from MicroK8s?

There may be something particular to MicroK8s?

And/or, someone there may have tried with GCR.

[mikebrow]

Others reporting it does work.. https://github.com/rancher/k3s/issues/1610

maybe I need to get a gcr account or try it against a service account

[mikebrow]

IIUC dockerd leverages containerd and I'm able to find 2 containerd processes running (one for Docker and the other for MicroK8s). Unfortunately, while the MicroK8s configuration uses containerd.toml to define these registries and auths, I'm unable to correlate this to the containerd instance run by Docker which works with the GCR registry; if I docker login, that config is stored in config.json and doesn't appear to be reflected explicity in the containerd.toml.

Perhaps I should ping the comment you cited from MicroK8s?

There may be something particular to MicroK8s?

And/or, someone there may have tried with GCR.

ah.. yeah while you can run with two containerd's you only need one containerd running... That should not cause you any problems...

containerd/cri uses the docker resolver code to make the connection.. hoping this is still just a config issue..

[mikebrow]

'{  "type": "service_account",  "project_id": "XXXX",  "private_key_id": "XXXX",  ...}'

note the single quotes due the to double quotes inside..

[DazWilkin]

Hmmm... the issue says that it doesn't work quoting this way (which is what I tried).

`{ "type": "service_account", ...}

It may be that I must escape the string:

"{ \"type\": \"service_account\", ...}

Still (!) I should not need to use a service account.

I should be able to authenticate using a access token.

I can create a service account for you to use with my registry?

[mikebrow]

yeah no.. I was just pointing out that these folks reported it working, in this fashion so there's hope.

[DazWilkin]

I'm going to try it on a different machine that's running MicroK8s too.

Please email me my handle at gmail dot com and I'll reply you the service account credentials.

I've given the account pull (and list only not push).

[mikebrow]

i'm going to go dig some holes in the back yard :-) will put gcr auth testing on the todo list for containerd/cri integration test unless you can also report it working, I need a week-end off, cheers

[DazWilkin]

Thanks for your help! Have fun! :-)

[mikebrow]

sure np it should be as simple as configure for gcr and run crictl to pull

[mikebrow]

ikr :-)

[mikebrow]

^ that way would eliminate anything microk8s is doing

[DazWilkin]

Escaping the password value was not the issue.

Abandoning for the weekend.

[DazWilkin]

I downloaded a standalone containerd (1.4.0-beta), applied the configurations I've been trying.

Same-same :-(

[DazWilkin]

OK.... Something curious.

I'm able to interact with ctr through MicroK8s and get more useful debugging:

sudo microk8s ctr --debug images pull gcr.io/[[PROJECT]]/hello-world:latest

I'm able to use this to pull e.g. public docker images successfully.

If I try to pull from GCR:

DEBU[0000] fetching                                      image="gcr.io/[[PROJECT]]/hello-world:latest"
DEBU[0000] resolving                                    
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:08 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:08 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:08 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0000] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:09 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0001] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:09 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] Unauthorized                                  header="Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull""
DEBU[0001] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *] Authorization:[Bearer REDACTED]] request.method=HEAD url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
DEBU[0001] fetch response received                       response.headers=map[Accept-Ranges:[none] Alt-Svc:[h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"] Cache-Control:[private] Content-Type:[application/json] Date:[Mon, 18 May 2020 17:32:09 GMT] Docker-Distribution-Api-Version:[registry/2.0] Server:[Docker Registry] Vary:[Accept-Encoding] Www-Authenticate:[Bearer realm="https://gcr.io/v2/token",service="gcr.io",scope="repository:[[PROJECT]]/hello-world:pull"] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]] status="401 Unauthorized" url="https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest"
ctr: failed to resolve reference "gcr.io/[[PROJECT]]/hello-world:latest": unexpected status code https://gcr.io/v2/[[PROJECT]]/hello-world/manifests/latest: 401 Unauthorized

I've redacted the tokens but they change each time which is OK except I should be able to base64 decode these and see something of the form:

_json_key:{ "type": "service_account", "project_id": .... }

But I get a bunch of binary data beginning 0x0090 0x03e6 -xfd83 ... (AJAD5v...)

[DazWilkin]

And:

sudo microk8s ctr version
Client:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

Server:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

[mikebrow]

it worked for me:

*** create gcp account with gcr, do all the steps to enable it to receive a
pushed image, including the download of a new _json_key (for a new
service account user..)... Then:

docker login -u _json_key -p "$(cat key.json)" gcr.io
docker push gcr.io/test-registry-277619/busybox
docker logout gcr.io

*** generate a single line for the key file generated by gcr:

jq -c . key.json

*** Edit the /etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]
  [plugins.cri.registry.mirrors."gcr.io"]
    endpoint = ["https://gcr.io"]
    [plugins.cri.registry.auths]
[plugins.cri.registry.auths."https://gcr.io"]
  username = "_json_key"
  password = 'output from jq'

*** restart containerd
mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ service containerd stop
mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ service containerd start

*** pull the image with crictl:

mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ sudo crictl pull gcr.io/test-registry-277619/busybox
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},Auth:nil,SandboxConfig:nil,} 
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,} 
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42

[mikebrow]

mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ sudo crictl rmi gcr.io/test-registry-277619/busybox
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] User specified image to be removed: gcr.io/test-registry-277619/busybox 
DEBU[0000] ImageStatusRequest: &ImageStatusRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},Verbose:false,} 
DEBU[0000] ImageStatusResponse: &ImageStatusResponse{Image:&Image{Id:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,RepoTags:[gcr.io/test-registry-277619/busybox:latest],RepoDigests:[gcr.io/test-registry-277619/busybox@sha256:a7766145a775d39e53a713c75b6fd6d318740e70327aaa3ed5d09e0ef33fc3df],Size_:762868,Uid:nil,Username:,},Info:map[string]string{},} 
DEBU[0000] RemoveImageRequest: &RemoveImageRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},} 
DEBU[0000] RemoveImageResponse: &RemoveImageResponse{}  
Deleted: gcr.io/test-registry-277619/busybox:latest
mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ sudo crictl pull gcr.io/test-registry-277619/busybox
DEBU[0000] get image connection                         
DEBU[0000] connect using endpoint 'unix:///run/containerd/containerd.sock' with '3s' timeout 
DEBU[0000] connected successfully using endpoint: unix:///run/containerd/containerd.sock 
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:gcr.io/test-registry-277619/busybox,},Auth:nil,SandboxConfig:nil,} 
DEBU[0001] PullImageResponse: &PullImageResponse{ImageRef:sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42,} 
Image is up to date for sha256:78096d0a54788961ca68393e5f8038704b97d8af374249dc5c8faec1b8045e42

[mikebrow]

@DazWilkin can you try with crictl? Let's try to minimize the surface area..

[DazWilkin]

Exciting!

I'd not actually tried using just username and password! Otherwise, the config matches.

I generally don't interact with these daemons directly so, my apologies for not being familiar with their tools either. I will work out how to use crictl

I'm walking my dog but will try this when home.

Thank you!

[mikebrow]

If you don't already have it... you could just clone this repo https://github.com/kubernetes-sigs/cri-tools then run make && make install. Otherwise I'd be surprised if the above config hack doesn't work for you.

[mikebrow]

my config for crictl.. in case you already have it and it's not configured to use containerd by default:

mike@mike-VirtualBox:~/go/src/github.com/kubernetes-sigs/cri-tools$ cat /etc/crictl.yaml runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: "" timeout: 3 debug: true

[DazWilkin]

Hmmm :-(

I'd been using:

[plugins.cri.registry.auths."gcr.io"]

NOTE this mirrors [plugins.cri.registry.mirrors."gcr.io"]

But, crictl works if this is:

[plugins.cri.registry.auths."https://gcr.io"]

And, per your repro, I only used the username and password values:

sudo crictl --runtime-endpoint=unix:/var/snap/microk8s/common/run/containerd.sock pull gcr.io/${PROJECT}/hello-world:latest
Image is up to date for sha256:bf756fb1ae65adf866bd8c456593cd24beb6a0a061dedf42b26a993176745f6b

However :-(

sudo microk8s ctr --debug images pull gcr.io/${PROJECT}/hello-world:latest

Still 401s :-(

[DazWilkin]

In other good news, this mechanism also works using:

username = "oauth2accesstoken"
password = "value from $(gcloud auth print-access-token)"

Which is preferable and further validates that this mechanism is good.

Very curiously (!) it also works in Kubernetes:

kubectl run hello-world --image=gcr.io/${PROJECT}/hello-world:latest
kubectl describe pod/hello-world
Successfully assigned default/hello-world to hp-pavilion
Pulling image "gcr.io/${PROJECT}/hello-world:latest"
Successfully pulled image "gcr.io/${PROJECT}/hello-world:latest"
Created container hello-world
Started container hello-world

So, my original problem is solved except I now don't understand why this doesn't work:

sudo microk8s ctr --debug images pull gcr.io/${PROJECT}/hello-world:latest

[mikebrow]

ctr isn't using containerd/cri so the CRI config won't modify how ctr tries to pull

[mikebrow]

containerd/cri is a plugin that operates via CRI service requests then calls to containerd to execute

crictl uses CRI services ..

containerd/CRI uses the config ^ above to generate the pull request with auth

[mikebrow]

i'm not familiar with microk8s code..

[mikebrow]

At some point we'll be refactoring the way we configure registry access, and that will include moving this config into containerd proper so ctr pulls would work with the same config.

[DazWilkin]

OK .... Well, thank you very much for your help!

To conclude:

• The configuration is as you show
• Albeit curiously while mirrors keys on gcr.io, auths keys on https://gcr.io
• username and password are required (auth and identitytoken are not)
• For service accounts _json_key and some variant of:
  • cat key.json | jq -c .
  • cat key.json | tr '\n' ' ' | sed 's|\"|\\"|'
• For access tokens oauth2accesstoken and $(gcloud auth print-access-token)

It appears that it's not possible to use auth only and the Docker equivalents:

printf "_json_key:%s" $(cat thalia.json | jq -c .) \
| base64 --wrap=0

printf "oauth2accesstoken:%s" $(gcloud auth print-access-token) \
| base64 --wrap=0

[DazWilkin]

You've been very helpful and patient.

I appreciate the education on containerd and cri

I didn't realize the crictl is a tool recommended for Kubernetes node debugging and will now use it.

Thank you!

[mikebrow]

sure np.. I haven't played with GCP/GCR so learned some things. I'll draft some PRs maybe make this a bit easier, esp. the docs. FYI kublet uses crictl to integrate with dockershim, containerd, and cri-o. So a little more than just a node debugging tool. Not an end user tool.. by any means but supported nonetheless.

[DazWilkin]

Popping back from this rabbit hole...

I'm going to update the Krustlet team on how GCR can be used (as well as Azure Container Registry) for OCI images.

Krustlet doesn't support imagePullSecret and so it's necessary to configure CRI as shown here.

[mikebrow]

@DazWilkin Thx.. @SteveLasker just an FYI since ACR use pattern was mentioned above for pulling container images(artifacts etc.) with auth setup via containerd/cri+krustlet+microk8s