docker / cli

The Docker CLI
Apache License 2.0
4.87k stars 1.92k forks source link

Docker Containerd snapshotter Overlayfs, docker save fails for image pulled using --platform=linux/arm64 on amd64 host. #5476

Open chetanshivaji opened 3 days ago

chetanshivaji commented 3 days ago

Description

docker save fails for image pulled using --platform=linux/arm64 on amd64 host.

  1. I pulled image as, docker pull --platform=linux/arm64 nginx:latest

docker images --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED       SIZE
nginx        latest    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3   5 weeks ago   273MB

2.

docker save -o nginx.tar sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3**
Error response from daemon: unable to create manifests file: content digest sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: not found

Not able to do docker save because of above error

3. Extra info, Although, docker save works with arm64 arch image if image is pulled using mainfest digest as below,

docker pull nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171

docker images --no-trunc
REPOSITORY   TAG       IMAGE ID                                                                  CREATED       SIZE
nginx        <none>    sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171   5 weeks ago   273MB

 docker save -o nginx.tar sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 - success

Reproduce

Same as above description.

Expected behavior

No response

docker version

docker version
Client:
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:13:00 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:13:08 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.8
  GitCommit:        7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[root@localhost sha256]#

docker info

# docker info
Client:
 Version:    25.0.3
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker Application (Docker Inc.)
    Version:  v0.8.0
    Path:     /usr/libexec/docker/cli-plugins/docker-app
  buildx: Build with BuildKit (Docker Inc.)
    Version:  v0.3.1-tp-docker
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: 25.0.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.4.17-2136.323.8.2.el7uek.x86_64
 Operating System: Oracle Linux Server 7.9
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.544GiB
 Name: localhost.localdomain
 ID: 0844a04b-e797-4332-881c-74867bca5af0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTPS Proxy: http://10.115.27.54:3128/
 No Proxy: localhost,127.0.0.1,localaddress,.localdomain.com
 Username: mchetan.more@gmail.com
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

[root@localhost sha256]#

Additional Info

No response

thaJeztah commented 2 days ago

Thanks for reporting! Yes, this is needs some work. I was curious if this was fixed on current master of the engine, and if PR would have fixed it;

But it looks like, while the error is slightly different 27.3 and "master" (looks like it now includes a NotFound prefix; possibly originating from containerd error-defs), the issue is still the same;

docker pull --platform=linux/s390x nginx:latest
latest: Pulling from library/nginx
a95d04829aec: Download complete
42b2ec246294: Download complete
95fe27c895a8: Download complete
6982f08f280c: Download complete
b2fff712f8b4: Download complete
1cdb42c01f48: Download complete
f9cf33a9c2ac: Download complete
Digest: sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

docker image ls --tree
WARNING: This is an experimental feature. The output may change and shouldn't be depended on.

IMAGE                   ID             DISK USAGE   CONTENT SIZE   IN USE
nginx:latest            04ba374043cc        246MB         65.3MB
├─ linux/s390x          0fa1ac852fea        246MB         65.3MB
├─ linux/amd64          88a0a069d5e9           0B             0B
├─ linux/arm/v5         6289301f3b91           0B             0B
├─ linux/arm/v7         d719acf8bcaf           0B             0B
├─ linux/arm64/v8       9f661996f4d1           0B             0B
├─ linux/386            67e126c43a8d           0B             0B
├─ linux/mips64le       83277e45a916           0B             0B
└─ linux/ppc64le        964b97f05849           0B             0B
docker save -o nginx.tar nginx:latest
Error response from daemon: unable to create manifests file: NotFound: content digest sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171: not found

daemon logs (current master) look like;

DEBU[2024-09-24T18:49:43.377261423Z] Calling HEAD /_ping                           spanID=c4ad8816f0c9fe7c traceID=d6ece5243ab000edd0dee3f6857257da
DEBU[2024-09-24T18:49:43.378616590Z] Calling GET /v1.48/images/get?names=nginx%3Alatest  spanID=0d7b5e44f6d43c8a traceID=d9a86fa6439b281f1f9320fbb2ffdcf8
DEBU[2024-09-24T18:49:43.385920632Z] export requested                              img="{docker.io/library/nginx:latest map[] {application/vnd.oci.image.index.v1+json sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 10272 [] map[] [] <nil> } 2024-09-24 18:46:01.821534085 +0000 UTC 2024-09-24 18:46:01.821534085 +0000 UTC}" name="nginx:latest" resolveErr="<nil>" spanID=0d7b5e44f6d43c8a specificDigestResolved=false traceID=d9a86fa6439b281f1f9320fbb2ffdcf8
DEBU[2024-09-24T18:49:43.401433173Z] export image                                  name="docker.io/library/nginx:latest" spanID=0d7b5e44f6d43c8a target="{application/vnd.oci.image.index.v1+json sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 10272 [] map[] [] <nil> }" traceID=d9a86fa6439b281f1f9320fbb2ffdcf8

The --platform flag (from https://github.com/docker/cli/pull/5331) helps in this case, but either the error message should be better (i.e., "we have the image, but not in the default platform"), or it should export only the available one;

docker save -o nginx.tar --platform=linux/s390x nginx:latest

ls -la nginx.tar
-rw------- 1 root root 65330176 Sep 24 18:47 nginx.tar
thaJeztah commented 2 days ago

Also tried to find what the sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 digest refers to, and it looks indeed the default (arm64) image;

docker buildx imagetools inspect nginx:latest
...
...
  Name:        docker.io/library/nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171
  MediaType:   application/vnd.oci.image.manifest.v1+json
  Platform:    linux/arm64/v8
  Annotations:
    org.opencontainers.image.url:             https://hub.docker.com/_/nginx
    org.opencontainers.image.version:         1.27.1
    com.docker.official-images.bashbrew.arch: arm64v8
    org.opencontainers.image.base.digest:     sha256:ace984fea4de21d4ad6d0047c3c6ab7ade9fcd9dee3e44c5fe6349d7dd063d86
    org.opencontainers.image.base.name:       debian:bookworm-slim
    org.opencontainers.image.created:         2024-09-05T12:38:34Z
    org.opencontainers.image.revision:        e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a
    org.opencontainers.image.source:          https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian

Full output;

```bash docker buildx imagetools inspect nginx:latest Name: docker.io/library/nginx:latest MediaType: application/vnd.oci.image.index.v1+json Digest: sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3 Manifests: Name: docker.io/library/nginx:latest@sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/amd64 Annotations: org.opencontainers.image.created: 2024-09-04T23:10:40Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 com.docker.official-images.bashbrew.arch: amd64 org.opencontainers.image.base.digest: sha256:903d3225acecaa272bbdd7273c6c312c2af8b73644058838d23a8c9e6e5c82cf org.opencontainers.image.base.name: debian:bookworm-slim Name: docker.io/library/nginx:latest@sha256:a6c3d2371006c10c41f4d3be8db4561b8f21c7cda4e8a3ab464521a00c336826 MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: com.docker.official-images.bashbrew.arch: amd64 vnd.docker.reference.digest: sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e vnd.docker.reference.type: attestation-manifest Name: docker.io/library/nginx:latest@sha256:6289301f3b918b98577fe0f80c13761aae5791b0059ed9e1d554edeae7c54b82 MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/arm/v5 Annotations: org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.0 com.docker.official-images.bashbrew.arch: arm32v5 org.opencontainers.image.base.digest: sha256:aef22c7b68e2f7707fabdff982d4d4928ca261775bae0ddb7f7f8358ed0ef10f org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-08-13T10:44:11Z org.opencontainers.image.revision: a6f7d140744f8b15ff4314b8718b3f022efc7f43 org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#a6f7d140744f8b15ff4314b8718b3f022efc7f43:mainline/debian Name: docker.io/library/nginx:latest@sha256:3e576b43defa219d279961a042477c8df1b2e46d15ec8cd98827eae750e33fa6 MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: com.docker.official-images.bashbrew.arch: arm32v5 vnd.docker.reference.digest: sha256:6289301f3b918b98577fe0f80c13761aae5791b0059ed9e1d554edeae7c54b82 vnd.docker.reference.type: attestation-manifest Name: docker.io/library/nginx:latest@sha256:d719acf8bcaf35a0a81288ea2a60a0fcab0b4cd65a0b6f68cadb060c5861f16e MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/arm/v7 Annotations: org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 com.docker.official-images.bashbrew.arch: arm32v7 org.opencontainers.image.base.digest: sha256:756f2240fee673990dccc7bdd288792e149ce50072d5cd29d328e4cd80aa6979 org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-09-05T04:28:40Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian Name: docker.io/library/nginx:latest@sha256:3a254dd24b20ecb921ff418f880892562a5d11a4de7e08465ab8296e4bc6a327 MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: com.docker.official-images.bashbrew.arch: arm32v7 vnd.docker.reference.digest: sha256:d719acf8bcaf35a0a81288ea2a60a0fcab0b4cd65a0b6f68cadb060c5861f16e vnd.docker.reference.type: attestation-manifest Name: docker.io/library/nginx:latest@sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/arm64/v8 Annotations: org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 com.docker.official-images.bashbrew.arch: arm64v8 org.opencontainers.image.base.digest: sha256:ace984fea4de21d4ad6d0047c3c6ab7ade9fcd9dee3e44c5fe6349d7dd063d86 org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-09-05T12:38:34Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian Name: docker.io/library/nginx:latest@sha256:0df868782671055c19e022b02b723375891f278fb785c0a89b4107b815c3ca1c MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: vnd.docker.reference.type: attestation-manifest com.docker.official-images.bashbrew.arch: arm64v8 vnd.docker.reference.digest: sha256:9f661996f4d1cea788f329b8145660a1124a5a94eec8cea1dba0d564423ad171 Name: docker.io/library/nginx:latest@sha256:67e126c43a8d738f2d05a7a7baa94a26b4d3f3f4cfb70b1aa49a52e700ef0709 MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/386 Annotations: org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 com.docker.official-images.bashbrew.arch: i386 org.opencontainers.image.base.digest: sha256:a09e6f30093763451525093f6c1f425fcb751de9afc5c7944a28e4e48250f4c9 org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-09-05T00:14:49Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian Name: docker.io/library/nginx:latest@sha256:96d7658c6592eee20fcb834cc3319cb7cbdaf487614132c3ff4da35b63e3baa9 MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: vnd.docker.reference.type: attestation-manifest com.docker.official-images.bashbrew.arch: i386 vnd.docker.reference.digest: sha256:67e126c43a8d738f2d05a7a7baa94a26b4d3f3f4cfb70b1aa49a52e700ef0709 Name: docker.io/library/nginx:latest@sha256:83277e45a9163e419c5e676f23498233b4e8f283daed1174db46a6e2d7860efa MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/mips64le Annotations: org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-09-05T11:19:25Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 com.docker.official-images.bashbrew.arch: mips64le org.opencontainers.image.base.digest: sha256:460b58fe46216c353ee64696b20a853678b1a08da92a64a1caf5e6faf997b329 Name: docker.io/library/nginx:latest@sha256:6f8fb2d834e6f7d91039dd2ddb653b809cfdb60f2646d839d17c714d8a2e2b9b MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: com.docker.official-images.bashbrew.arch: mips64le vnd.docker.reference.digest: sha256:83277e45a9163e419c5e676f23498233b4e8f283daed1174db46a6e2d7860efa vnd.docker.reference.type: attestation-manifest Name: docker.io/library/nginx:latest@sha256:964b97f05849b3ca619df5bea76d98b53196260cf8cd9b6c03ccbb8393dc8613 MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/ppc64le Annotations: com.docker.official-images.bashbrew.arch: ppc64le org.opencontainers.image.base.digest: sha256:f39559bf405679e55f74c361f383e0cfe19b01934a53eca2149ddd966604293b org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-09-05T00:43:55Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 Name: docker.io/library/nginx:latest@sha256:a72749cca16dd47485c9220a98b74b04d6f564ef8ec6c189e85c56e04c7e4dc6 MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: com.docker.official-images.bashbrew.arch: ppc64le vnd.docker.reference.digest: sha256:964b97f05849b3ca619df5bea76d98b53196260cf8cd9b6c03ccbb8393dc8613 vnd.docker.reference.type: attestation-manifest Name: docker.io/library/nginx:latest@sha256:0fa1ac852feac85b15bdec82773c1b19eafdde054536e2f7f4c05cecbff96433 MediaType: application/vnd.oci.image.manifest.v1+json Platform: linux/s390x Annotations: com.docker.official-images.bashbrew.arch: s390x org.opencontainers.image.base.digest: sha256:d11dd86eb4857cdeac7b99af3884fa13e8a3d18ce550f77cee3cca64a735f233 org.opencontainers.image.base.name: debian:bookworm-slim org.opencontainers.image.created: 2024-09-05T02:49:36Z org.opencontainers.image.revision: e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a org.opencontainers.image.source: https://github.com/nginxinc/docker-nginx.git#e78cf70ce7b73a0c9ea734c9cf8aaaa283c1cc5a:mainline/debian org.opencontainers.image.url: https://hub.docker.com/_/nginx org.opencontainers.image.version: 1.27.1 Name: docker.io/library/nginx:latest@sha256:7df7564b1cfdd1f098b8ffb3d91a91ed248a1d54826a8260b52ff225b70cc353 MediaType: application/vnd.oci.image.manifest.v1+json Platform: unknown/unknown Annotations: com.docker.official-images.bashbrew.arch: s390x vnd.docker.reference.digest: sha256:0fa1ac852feac85b15bdec82773c1b19eafdde054536e2f7f4c05cecbff96433 vnd.docker.reference.type: attestation-manifest ```
chetanshivaji commented 1 day ago

Thanks for the details @thaJeztah.

This seems to be an issue with image pulls with --platform mismatching to host arch.

docker inspect 04ba374043cc|grep -ani arch

74:        "Architecture": "386",
docker images --no-trunc

REPOSITORY            TAG       IMAGE ID                                                                  CREATED             SIZE
nginx                 latest    sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3   5 weeks ago         252MB

docker save -o nginx386arch.tar sha256:04ba374043ccd2fc5c593885c0eacddebabd5ca375f9323666f28dfd5a9710e3

Error response from daemon: unable to create manifests file: content digest sha256:88a0a069d5e9865fcaaf8c1e53ba6bf3d8d987b0fdc5e0135fec8ce8567d673e: not found
thaJeztah commented 1 day ago

@chetanshivaji yes, correct; the issue is that the image pulled does not match the default platform. When running docker save, the initial part "succeeds" (multi-arch image exists), but it then tries to save the default (native) platform, and that is missing.

So at least the error-message should be more clear, but we should also look if the request is non-ambiguous (i.e., image exists, and there's only a single platform locally -> in that case we could save the image, regardless if it's the default platform).