Closed bmelbourne closed 1 month ago
Here is the output from ctr
which also reports the correct image digest...
$ sudo ctr --namespace k8s.io images list | grep kube-apiserver
registry.k8s.io/kube-apiserver:v1.28.4 application/vnd.docker.distribution.manifest.list.v2+json sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb 30.1 MiB linux/amd64,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
@saschagrunert @haircommander is this related to the image id work?
@kannon92 @bmelbourne this is likely an issue with containerd, since you can reproduce it only with nerdctl
and crictl
.
You probably want to move this issue to the containerd repo.
/cc @mikebrow
@saschagrunert
In the description you'll see that the issue is specific to crictl
only, both nerdctl
and ctr
report the correct pullable image digest for registry.k8s.io/kube-apiserver:v1.28.4
.
/cc @mikebrow
@bmelbourne but the underlying runtime is containerd with a CRI middle layer, right? Means that containerd or the CRI shim needs to take care of the result, not crictl.
@saschagrunert not sure I agree, everything provided in the description has been generated from the same containerd
runtime running on the same AWS cluster node running Ubuntu 22.04, and both nerdctl
and ctr
are reporting the correct pullable image digest.
If you look closely at the following output of the nerdctl pull
, it appears crictl
is using the config-sha256
hash and not the index-sha256
hash which would provide the correct image digest for any CRI tool to pull from registry.k8s.io
.
registry.k8s.io/kube-apiserver@sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb: done |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:a4c3e6bec39f5dcb221a2f08266513ab19b7d977ccc76a0bcaf04d4935ac0fb2: done |++++++++++++++++++++++++++++++++++++++|
config-sha256:04b4c447bb9d4840af3bf7e836397379d65df87c86e55dcd27f31a8d11df2419: done |++++++++++++++++++++++++++++++++++++++|
@kannon92 @mikebrow what are your thoughts?
We get the images from the ListImages RPC and then normalize the digests by just using the first one:https://github.com/kubernetes-sigs/cri-tools/blob/17b4dd65d660fec94d7a5a070e3e89ef640f1087/cmd/crictl/image.go#L626-L630
Maybe we should take multiple digests into account, but what's the value of repoDigests[0]
is in the hands of containerd.
Thanks for the specific code snippet, I'll compare this with how nerdctl
and ctr
are retrieving the image digest, and then raise the issue with the containerd
project for their feedback.
Maybe something in recent containerd
releases, and/or the OCI image spec means repoDigests[0]
is being assigned the config-sha256
hash.
Hello. In your example for crictl images you are looking at the image ID not the image ref.... At the moment (the last 6y), yes the image id we are showing on image pull / list etc. for use by the cri apis, for example to say remove the cri image.., is the config digest.. But that is just an id to reference our internal cri meta for the actual cri image. confusing if you think it's the digest of the image/index..
What you want, I believe, is repo digest:
(base) root@ubnt:~# crictl images --digests
IMAGE TAG DIGEST IMAGE ID SIZE
docker.io/library/busybox 1.35.0 02289a9972c50 0c00acac9c279 2.22MB
docker.io/library/nginx latest ea97e6aace270 247f7abff9f70 70.5MB
registry.k8s.io/coredns/coredns v1.11.1 1eeb4c7316bac cbb01a7bd410d 18.2MB
registry.k8s.io/pause 3.6 3d380ca886454 6270bb605e12e 302kB
registry.k8s.io/pause 3.9 7031c1b283388 e6f1816883972 322kB
(base) root@ubnt:~# crictl images --digests -v
ID: sha256:0c00acac9c2794adfa8bb7b13ef38504300b505a043bf68dff7a00068dcc732b
RepoTags: docker.io/library/busybox:1.35.0
RepoDigests: docker.io/library/busybox@sha256:02289a9972c5024cd2f083221f6903786e7f4cb4a9a9696f665d20dd6892e5d6
Size: 2223917
FYI cool tool that Derek put together:
ctr -n k8s.io images i docker.io/library/busybox:1.35.0 --content
I get the same results with dockerd, so I don't think that it is related to containerd.
It has another intesting quirk, in that filtering the images removes the digests...
$ docker images --digests busybox:latest
REPOSITORY TAG DIGEST IMAGE ID CREATED SIZE
busybox latest <none> 3f57d9401f8d 6 weeks ago 4.26MB
$ docker images --digests | grep busybox | grep latest
busybox latest sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74 3f57d9401f8d 6 weeks ago 4.26MB
This behaviour also spreads to cri-dockerd, so filtered images end up with <none>
$ sudo crictl images --digests busybox:latest
busybox latest <none> 3f57d9401f8d4 4.26MB
$ sudo crictl images --digests | grep busybox | grep latest
busybox latest 6d9ac9237a84a 3f57d9401f8d4 4.26MB
But otherwise, it seems to be about the Id
versus the RepoDigest
, as mentioned.
"Id": "sha256:3f57d9401f8d42f986df300f0c69192fc41da28ccc8d797829467780db3dd741",
"RepoTags": [
"busybox:latest"
],
"RepoDigests": [
"busybox@sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74"
],
Images that are built locally only have an Id, they won't get a digest until they are pushed to a repo.
"RepoTags": [
"myimage:latest"
],
"RepoDigests": [],
I've created a clean standalone AWS node with no pre-existing running Kubernetes cluster, and I believe the issue relates to how the ctr images pull
command is not setting the image digest correctly.
Based on the following output, I'd appreciate an expert review to confirm my assumption before I close this ticket, and raise the issue with the containerd
project?
$ sudo kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.7", GitCommit:"c8dcb00be9961ec36d141d2e4103f85f92bcf291", GitTreeState:"clean", BuildDate:"2024-02-14T10:39:01Z", GoVersion:"go1.21.7", Compiler:"gc", Platform:"linux/arm64"}
$ sudo containerd --version
containerd github.com/containerd/containerd v1.7.13 7c3aca7a610df76212171d200ca3811ff6096eb8
$ sudo ctr --version
ctr github.com/containerd/containerd v1.7.13
$ sudo crictl --version
crictl version v1.28.0
$ sudo kubeadm config images pull --kubernetes-version=v1.28.4
[config/images] Pulled registry.k8s.io/kube-apiserver:v1.28.4
[config/images] Pulled registry.k8s.io/kube-controller-manager:v1.28.4
[config/images] Pulled registry.k8s.io/kube-scheduler:v1.28.4
[config/images] Pulled registry.k8s.io/kube-proxy:v1.28.4
[config/images] Pulled registry.k8s.io/pause:3.9
[config/images] Pulled registry.k8s.io/etcd:3.5.10-0
[config/images] Pulled registry.k8s.io/coredns/coredns:v1.10.1
$ sudo crictl images --digests | grep kube-apiserver
registry.k8s.io/kube-apiserver v1.28.4 5b28a364467cf 04b4c447bb9d4 31.6MB
$ sudo ctr -n k8s.io images list | grep kube-apiserver
registry.k8s.io/kube-apiserver:v1.28.4 application/vnd.docker.distribution.manifest.list.v2+json sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb 30.1 MiB linux/amd64,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
registry.k8s.io/kube-apiserver@sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb application/vnd.docker.distribution.manifest.list.v2+json sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb 30.1 MiB linux/amd64,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
$ sudo crictl rmi --prune
Deleted: registry.k8s.io/kube-apiserver:v1.28.4
Deleted: registry.k8s.io/kube-controller-manager:v1.28.4
Deleted: registry.k8s.io/kube-scheduler:v1.28.4
Deleted: registry.k8s.io/kube-proxy:v1.28.4
Deleted: registry.k8s.io/pause:3.9
Deleted: registry.k8s.io/etcd:3.5.10-0
Deleted: registry.k8s.io/coredns/coredns:v1.10.1
$ sudo ctr -n k8s.io images pull --hosts-dir /etc/containerd/certs.d registry.k8s.io/kube-apiserver:v1.28.4
registry.k8s.io/kube-apiserver:v1.28.4: resolved |++++++++++++++++++++++++++++++++++++++|
index-sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb: done |++++++++++++++++++++++++++++++++++++++|
...
elapsed: 1.1 s total: 30.1 M (27.4 MiB/s)
unpacking linux/arm64/v8 sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb...
done: 867.437748ms
$ sudo crictl images --digests
IMAGE TAG DIGEST IMAGE ID SIZE
registry.k8s.io/kube-apiserver v1.28.4 <none> 04b4c447bb9d4 31.6MB
$ sudo ctr -n k8s.io images list
REF TYPE DIGEST SIZE PLATFORMS
LABELS
registry.k8s.io/kube-apiserver:v1.28.4 application/vnd.docker.distribution.manifest.list.v2+json sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb 30.1 MiB linux/amd64,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
sha256:04b4c447bb9d4840af3bf7e836397379d65df87c86e55dcd27f31a8d11df2419 application/vnd.docker.distribution.manifest.list.v2+json sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb 30.1 MiB linux/amd64,linux/arm64,linux/ppc64le,linux/s390x io.cri-containerd.image=managed
You'll notice from the above output, ctr
lists the correct image info, tag and digest e.g. registry.k8s.io/kube-apiserver@sha256:5b28a364467cf7e134343bb3ee2c6d40682b473a743a72142c7bbe25767d36eb
when using kubeadm config images pull
to pull the registry.k8s.io
images, but when using ctr images pull
, the image digest is not fully formed and incorrectly shows sha256:04b4c447bb9d4840af3bf7e836397379d65df87c86e55dcd27f31a8d11df2419
for the image digest.
You say image digest, but reference the image id?
Yes, I didn't quite explain it properly, the ctr images pull
command is not populating the Image ID
correctly. The image path is missing and the image digest is incorrect.
Okay, that behaviour I can reproduce. Using ctr
(or nerdctl
) does not create the digests entry.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
/remove-lifecycle stale
I'm not sure if there is something we can fix here from the cri-tools perspective.
What happened:
When attempting to use a vulnerability scanner, such as
kubevuln
as part of the Kubescape security platform, thekube-apiserver v1.28.4
pod is not reporting the correct image digest and hence, the image cannot be pulled fromregistry.k8s.io
, scanned and verified.crictl v1.28.0
nerdctl v1.7.1
kube-apiserver v1.28.4
What you expected to happen:
Closer examination shows that
crictl
is incorrectly setting the image ID in the container runtime usingconfig-sha256
and notindex-sha256
which matches the image digest stored in the remote kubernetesregistry.k8s.io
image registry, and hence, security scanners such askubevuln
would be able to use the correct digest and pull the image successfully.How to reproduce it (as minimally and precisely as possible):
kubeadm
and pull the kubernetesv1.28.4
system imagesWhat happened
section to review thekube-apiserver
image digestAnything else we need to know?:
Environment:
cat /etc/os-release
):uname -a
):