Error nerdctl commit, "failed to create a tmp single-platform image..., content digest ... not found"

[jooseop]

Description

I am using Kubernetes, and the container runtime is containerd. I want to commit an image from within a pod.

Inside the pod, nerdctl is installed, and the pod has mounted the node's /run/containerd and /var/lib/containerd directories as hostPath volumes.

When I try to commit using nerdctl, I encounter the following error:

"time="2024-09-05T08:09:43Z" level=fatal msg="failed to create a tmp single-platform image \"registry:tag-tmp-reduced-platform\": content digest sha256:602d8ad51b8130f3fcd71cb936dea612ebc799666136abf2e5914585b3178a4a: not found" The registry and tag seem fine. I've also tried pulling the base image before running nerdctl. I already attempted using the --unpack=false flag as well.

Do you know of any solution for this issue?

Steps to reproduce the issue

No response

Describe the results you received and expected

complete commit

What version of nerdctl are you using?

nerdctl commit

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

Others

Host information

No response

[apostasie]

Possibly a duplicate of #827 ?

[jooseop]

"failed to create a tmp single-platform image" -> Solved issue, using the '--all-platforms' option

However, " content digest ... not found" still remains.

[apostasie]

@AkihiroSuda we need to fix this. We got many reports / variants of it.

I believe this is flaky - sometimes the Kube test would pass, sometimes it would fail.

Someone mentioned it could be containerd garbage collecting layers.

Tagging @lingdie who did some work already on this, and @fahedouch who might have an idea what's going on (IIRC he mentioned before that this was happening as well outside of kube?).

Anyone has a good lead to start investigating? Or at least a simpler way to repro without Kube?

[jooseop]

There are issues when committing a pod running an image built from a Dockerfile. Could the issue be caused by the image built from the Dockerfile? Committing a pod running an image from NVIDIA GPU Cloud worked fine

[apostasie]

There are issues when committing a pod running an image built from a Dockerfile. Could the issue be caused by the image built from the Dockerfile? Committing a pod running an image from NVIDIA GPU Cloud worked fine

Unfortunately, we reproduced with prebuilt images - so I would say that this is unlikely - but then again, we don't know at this point...

[lingdie]

I'm not sure, but I do know that crt can push images that have been committed using nerdctl.

[jooseop]

Steps

1. nerdctl --namespace k8s.io pull --unpack=false {pod image}
2. nerdctl --namespace k8s.io commit {nerdctl pod container id}
3. nerdctl push --all-platforms

Error

1. nerdctl push error message : content digest ... not found
   "ctr push" also same problem.
2. When there is an error, the content digest in the error message is not in the path (/var/lib/containerd/io.containerd.content.v1.content/blobs/sha256)
3. I check "ctr image check" option, Image imcompleted. but no error message "nerdctl pull"

   acr-0-jworker-6:/# ctr --namespace=k8s.io i check | grep ubun
   registry/custom_ubuntu-20.04.6:latest              application/vnd.docker.distribution.manifest.v2+json      sha256:77649be7ae3b6950e0d6383c23f251e15cf58624d5ff64b40e35572663ce8552 incomplete (10/11) 930.8 MiB/957.0 MiB true

Solution

1. ctr --namespace k8s.io image pull
2. nerdctl commit --namespace k8s.io
3. ctr push --all-platforms

Isn't there a problem with "nerdctl pull"?

[lingdie]

not nerdctl pull, bug containerd pull. see: https://github.com/containerd/containerd/issues/8973