Open alex-ioma opened 3 weeks ago
I'm having the same issue. It's not clear anymore from the documentation (https://github.com/containerd/containerd/blob/main/docs/cri/registry.md and https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration) how to add credentials for private registries. The documents mention deprecated configuration surfaces and even mention there won't be a way to configure authentication with plain text. I tried to read the code, but it's a rabbit hole.
What's the best way to configure private registry access for CRI?
us-central1-docker.pkg.dev
Hi folks, I believe the right way to configure this would be https://github.com/containerd/cri/blob/master/docs/registry.md#configure-registry-credentials-example---gcr-with-service-account-key-authentication ?
Hi folks, I believe the right way to configure this would be https://github.com/containerd/cri/blob/master/docs/registry.md#configure-registry-credentials-example---gcr-with-service-account-key-authentication ?
Thanks @neoaggelos but that is not ok.
The manual use of Service Account Key to generate a Token (and then use that token in the docker auth) is not a sustainable approach. This is because that token is a short-lived token that needs to be regenerated with a refresh-token via a oath approach. Such approach must be handled directly by the containerd process by using the Service Account Key (the key needs to be generated with the least possible permission on GPC IAM section, of course).
To be clear, Kubernetes is already able to manage such oauth token/refresh-token approach by providing GCP key as auth
parameter of the .dockerconfig secrets. But such special secret needs to be added to each namespace that needs to pull images from that registry. However, the same approach doesn't work if someone wants to implemented it at containerd "raw" level.
Different post online reports possible bug / issues with conatinerd itself that is not able to manage such oauth process but I'm not sure.
I was hoping to have at least a confirmation from the canonical team.
Hi @alex-ioma, thank you for elaborating on why this is not a sustainable approach.
Overall, indeed, this looks something containerd specific, and not a limitation or bug on the side of MicroK8s. I believe raising an issue at https://github.com/containerd/containerd might be more useful towards a resolution.
Summary
crictl
orctr
is not able to pull images from Artifact Registry via Service Account Key. The service account can correctly pull images if the related key is added as .dockerconfig secret within k8s but it does not work if added as a docker config file to the underlying containerd.crictl version v1.26.1 ctr github.com/k3s-io/containerd v1.7.11-k3s2
What Should Happen Instead?
Images are pulled.
Reproduction Steps
auth
e.g. via config.toml.tmpl
Introspection Report
I believe the issue is on how the auth is parsed and used. The key is a long-lived json-key file that should use an RSA key to retrieve a
oauth
JWT token via Bearer token from the googletoken_uri
. It looks like containerd is trying to get the token from the repositoryURI
Can you suggest a fix?
Not sure how to troubleshoot further but might be similar to issue #990
Are you interested in contributing with a fix?
Happy to test possible solution strategies.