Cannot commit and push container image from Kubernetes pods

nettoclaudio commented 2 years ago

Hello folks! :wave:

I'm experiencing a strange behavior while committing a container (from Kubernetes pod) and pushing it to some container registry after. I always get the same error from the push command: content digest sha256:898c46f3b1a1f39827ed135f020c32e2038c87ae0690a8fe73d94e5df9e6a2d6: not found.

I'm filing this issue here but there's a chance the problem is on containerd side because I've already seen it working in an earlier version. I'm trying to find out which version it stopped to work.

I'm still investigating but no idea so far... suggestions are welcome.

Steps to reproduce:

Create an arbitrary Kubernetes pod (for simplicity w/ just one container).
```
$ kubectl run --image tsuru/go:latest my-app -- sleep Inf
```
Execute arbitrary commands to generate changes in the container image (to be committed):
```
$ kubectl exec my-app -- mkdir -p /tmp/foo/bar
```

Commit the container to a local image:

$ CONTAINER_ID=$(kubectl get pods my-app -o 'jsonpath={ .status.containerStatuses[0].containerID }' | sed -E 's|(.+)://||g')
$ nerdctl --namespace k8s.io commit ${CONTAINER_ID} registry.example.com/my-app:v1

Push to local registry (no credentials needed):

$ nerdctl --namespace k8s.io push registry.example.com/my-app:v1
FATA[0000] failed to create a tmp single-platform image "registry.example.com/my-app:v1-tmp-reduced-platform": content digest sha256:898c46f3b1a1f39827ed135f020c32e2038c87ae0690a8fe73d94e5df9e6a2d6: not found

Environment information:

nerdctl: 0.17.0
containerd: v1.4.12 7b11cfaabd73bb80907dd23182b9347b4245eb5d
OS: Buildroot 2021.02.4 (from minikube)
Kernel: 4.19.202
Kubernetes version: v1.20.6
kubectl version: v1.20.6
Minikube: v1.25.1

junnplus commented 2 years ago

I can't reproduce this problem on containerd with latest and 1.5.10 version.

jun@lima-k8s-test:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
jun@lima-k8s-test:~$ containerd -v
containerd containerd.io 1.5.10 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc

In addition, I can't start the corresponding containerd version through minikube.

PS: containerd v1.4.x reached EOL.

junnplus commented 2 years ago

I closed it because containerd v1.4.x reached EOL.

Feel free to open it up if you have any other questions.

Nirmit1995 commented 1 year ago

I am also getting the same error for large images (mine was 7GB, small images were easily pushed)

FATA[0000] failed to create a tmp single-platform image "registry.example.com/my-app:0086bf16-21cb-477b-952c-13780780f597-tmp-reduced-platform": content digest sha256:5bed26d33875e6da1d9ff9a1054c5fef3bbeb22ee979e14b72acf72528de007b: not found

I have the following config- [ec2-user@ip]$ sudo /usr/local/bin/nerdctl version WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH Client: Version: v1.4.0 OS/Arch: linux/amd64 Git commit: 7e8114a82da342cdbec9a518c5c6a1cce58105e9 buildctl: Version:

Server: containerd: Version: 1.6.19 GitCommit: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f runc: Version: 1.1.4 GitCommit: 5fd4c4d144137e991c4acebb2146ab1483a97925

Please help in resolving this.

HFfleming commented 1 year ago

could you please reopen the issue, @junnplus thanks. I meet the same problem : I commit the image from a running container, then i need push the new image to my repo. when i use nerdctl -n k8s.io push xxx ,an error occurs: FATA[0000] failed to create a tmp single-platform image "xxx.com/hjmtest/nginx:hjm-tmp-reduced-platform": content digest sha256:550fe1bea624a5c62551cf09f3aa10886eed133794844af1dfb775118309387e: not found

here is my env info: nerdctl -v nerdctl version 1.4.0 containerd -v containerd github.com/containerd/containerd v1.6.14-42-g21f32b2c3 21f32b2c394bc1ccfbf1744876a0fcdd4ef2390d

HFfleming commented 1 year ago

could you please open this issue, @junnplus thanks. I meet the same problem : I commit the image from a running container, then i need push the new image to my repo. when i use nerdctl -n k8s.io push xxx ,an error occurs: FATA[0000] failed to create a tmp single-platform image "xxx.com/hjmtest/nginx:hjm-tmp-reduced-platform": content digest sha256:550fe1bea624a5c62551cf09f3aa10886eed133794844af1dfb775118309387e: not found

here is my env info: nerdctl -v nerdctl version 1.4.0 containerd -v containerd github.com/containerd/containerd v1.6.14-42-g21f32b2c3 21f32b2c394bc1ccfbf1744876a0fcdd4ef2390d

In addition: if i pull images from others, not commit a new image, nerdctl push xxxcan work.

ggerogery commented 1 year ago

I got the same issue but with finch. Solved with --platform arg: finch push xxx.dkr.ecr.us-east-1.amazonaws.com/infra/calico/apiserver:v3.25.1 --platform linux/amd64

baozaolaoba-top commented 10 months ago

In addition:

nerdctl version 1.5.0 containerd -v containerd github.com/containerd/containerd v1.7.1 1677a17964311325ed1c31e2c0a3589ce6d5c30d

harbor server with port: hub.local:5443

Zheaoli commented 10 months ago

WARN[0000] unable to determine buildctl version: exec: "buildctl": executable file not found in $PATH 
Client:
 Version:       v1.5.0
 OS/Arch:       linux/amd64
 Git commit:    b33a58f288bc42351404a016e694190b897cd252
 buildctl:
  Version:

Server:
 containerd:
  Version:      1.7.1+azure-1
  GitCommit:    1677a17964311325ed1c31e2c0a3589ce6d5c30d
 runc:
  Version:      1.1.7
  GitCommit:    860f061b76bb4fc671f0f9e900f7d80ff93d4eb7

could not produce this bug. would you guys mind to give me reproduce step with more detail? like what's registry you use

JCereal commented 8 months ago

Recently, my GKE cluster's worker node upgraded to v1.27.3-gke.100 which also upgraded containerd from 1.6.18 to 1.7.0. After the upgrade, I encountered the exact same issue.

Client:
 Version:   v1.7.0
 OS/Arch:   linux/amd64
 Git commit:    e674fe7ba6e49f12e88cd9c6c442e7ea5232502c
 buildctl:
  Version:  v0.12.3
  GitCommit:    438f47256f0decd64cc96084e22d3357da494c27

Server:
 containerd:
  Version:  1.7.0
  GitCommit:    1fbd70374134b891f97ce19c70b6e50c7b9f4e0d
 runc:
  Version:  1.1.10
  GitCommit:    v1.1.10-0-g18a0cb0f

Reproduce

The commit command: nerdctl -n k8s.io commit bbe68406a258 some.registry.com/my-app:0.1

The push command like: nerdctl -n k8s.io push some.registry.com/my-app:0.1

Output: FATA[0000] failed to create a tmp single-platform image "some.registry.com/my-app:0.1-tmp-reduced-platform": content digest sha256:c95ff01263cbbb536e71f8ae823d3e63f15f7a0f1ba9ecb7b3126a63654d2b23: not found.

Work around

Add --all-platforms option.

If I run the push command like: nerdctl -n k8s.io push --all-platforms some.registry.com/my-app:0.1, the image can be pushed if the registry is valid (btw I pushed to JFrog).

The option is added to avoid converter.Convert. Related code: https://github.com/containerd/nerdctl/blob/ce2f63d275c80f1ba2a3e2d9bfdc15ded3ff73c7/pkg/cmd/image/push.go#L98-L111

Other info

The issue is not registry related. I tried different registries, the same error is thrown, even with random invalid registry.

The container I am committing from is built on an Ubuntu machine with docker build . command, without specifying the platform as linux/amd64.

When it used to work with containerd 1.6.18, I can see the log level=info msg="pushing as a reduced-platform image.... The convert should be good then.

beyou923 commented 7 months ago

I started a container using the nydus format and then packaged the container into an image. However, I encountered the same problem when I tried to push this image.

Steps to reproduce:

Create a container with nydus image

nerdctl --snapshotter nydus run --cap-add=CAP_SYS_ADMIN  --net host -it  registry-test.xx.info/library/nydus-image:old

Commit the container to a local image:

nerdctl  commit ${CONTAINER_ID} registry-test.xx.info/library/nydus-image:new

Push to registry (no credentials needed):

nerdctl push registry-test.xx.info/library/nydus-image:new
FATA[0000] failed to create a tmp single-platform image " registry-test.xx.info/library/nydus-image:new-tmp-reduced-platform": content digest sha256:cfc7393224358fb100dabf40f7d59c2578c0a6a47781b1a6b56406922aeb6e95: not found

env info:

Kernel: 4.9.144
nerdctl version 1.6.2
containerd: v1.5.9
nydusd version: v2.2.3
containerd-nydus-grpc version: v0.13.3

Does the container launched by on-demand loading of images support commit and push operations?

iholo commented 5 months ago

I have the same problem on Azure

env info: kubernetes: v1.27.7 Kernel: 5.15.0-1053-azure nerdctl version 1.7.3 containerd: 1.7.1+azure-1

michaelmalice commented 1 month ago

Hi I'm having the same issue. Is there a specific version of containerd that I can use that doesn't have this problem?

hj1801 commented 3 weeks ago

I am encountering the same issue. It seems to occur from kubernetes 1.27 version, and it remains unresolved even when using the latest version of nerdctl (1.7.6). As someone else pointed out, the --all-platforms or --platform flag does not work while pushing. Looking at the issue, it seems that it starts from failing to save a committed image.

This has been an ongoing issue since the past and I am wondering if there have been any follow-ups on this matter.

containerd / nerdctl