containerd-integration: platform matching with partial platforms works different with containerd integration

thaJeztah commented 1 year ago

Description

I noticed this when trying to compare https://github.com/moby/moby/pull/44390 with the current behaviour, but then noticed that the containerd-integration acts different than before the integration.

Reproduce

When running a docker pull with a partial ARM platform (no variant specified), containerd integration pull all (32-bit) arm variants:

docker pull --platform=linux/arm hello-world
Using default tag: latest
e18f0a777aef: Exists
f130bd2d67e6: Exists
1ec996c686eb: Exists
docker.io/library/hello-world:latest

As can be seen above, it's pulling 3 images (effectively treating linux/arm as linux/arm/* (wildcard); the digests in the output above match the various linux/arm variants;

docker manifest inspect hello-world@sha256:e18f0a777aefabe047a671ab3ec3eed05414477c951ab1a6f352a06974245fe7
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 525,
         "digest": "sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 525,
         "digest": "sha256:7b8b7289d0536a08eabdf71c20246e23f7116641db7e1d278592236ea4dcb30c",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v5"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 525,
         "digest": "sha256:f130bd2d67e6e9280ac6d0a6c83857bfaf70234e8ef4236876eccfbd30973b1c",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 525,
         "digest": "sha256:432f982638b3aefab73cc58ab28f5c16e96fdb504e8c134fc58dff4bae8bf338",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },

However, after the pull completes, no image is stored:

docker image inspect hello-world
[]
Error: No such image: hello-world

docker image inspect docker.io/library/hello-world:latest
[]
Error: No such image: docker.io/library/hello-world:latest

Without containerd integration (docker 20.10 and the PR branch from https://github.com/moby/moby/pull/44390) only a single variant is pulled ("best match");

docker pull --platform=linux/arm hello-world
Using default tag: latest
latest: Pulling from library/hello-world
9b157615502d: Pull complete
Digest: sha256:e18f0a777aefabe047a671ab3ec3eed05414477c951ab1a6f352a06974245fe7
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

But, interestingly, it doesn't use v8, but uses v7 (I guess this may be intentional because the platform specified was arm, not arm64, and for arm64, the full platform is linux/arm64/v8;

docker image inspect --format '{{.Os}}/{{.Architecture}}/{{.Variant}}' hello-world
linux/arm/v7

Expected behavior

A single arch to be pulled (best match) and the image to be present in docker image ls.

docker version

Server: Docker Desktop 4.13.0 (89412)
 Engine:
  Version:          22.06.0-beta.0-407-g7a5cce156e.m
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.18.4
  Git commit:       7a5cce156e
  Built:            Thu Oct 13 08:09:41 2022
  OS/Arch:          linux/arm64
  Experimental:     true
 containerd:
  Version:          1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

(not relevant)

Additional Info

No response

tianon commented 1 year ago

In containerd's platform strings, linux/arm implies linux/arm/v7 (Edit: source: https://github.com/containerd/containerd/blob/8c5baf4ebb5a53911f6f8e54f931482794e01e12/platforms/database.go#L108-L109), and the three IDs you're seeing are not three images, but three objects for the one image (manifest list, image manifest, image config):

e18f0a777aef: Content-Type: application/vnd.docker.distribution.manifest.list.v2+json (sha256:e18f0a777aefabe047a671ab3ec3eed05414477c951ab1a6f352a06974245fe7)
f130bd2d67e6: Content-Type: application/vnd.docker.distribution.manifest.v2+json (sha256:f130bd2d67e6e9280ac6d0a6c83857bfaf70234e8ef4236876eccfbd30973b1c)
1ec996c686eb: Content-Type: application/vnd.docker.container.image.v1+json (sha256:1ec996c686eb87d8f091080ec29dd1862b39b5822ddfd8f9a1e2c9288fad89fe)

I'm actually a little surprised we don't see a fourth ID for the layer blob itself (sha256:9b157615502ddff86482f7fe2fa7a074db74a62fce12b4e8507827ac8f08d0ce). :thinking:

thaJeztah commented 1 year ago

Oh! I completely missed your message here, @tianon 😂 was just discussing this with @vvoland, and it looks like the pulls that are shown may indeed be the manifests, not the image as a whole. So this may be due to the changes made to match containerd's matching (which recurses manifests prior to matching?)

time docker pull --platform=linux/arm hello-world
Using default tag: latest
e18f0a777aef: Download complete
f130bd2d67e6: Download complete
1ec996c686eb: Download complete
9b157615502d: Download complete
docker.io/library/hello-world:latest

________________________________________________________
Executed in    6.41 secs      fish           external
usr time   60.29 millis    0.12 millis   60.17 millis
sys time   36.71 millis    2.15 millis   34.56 millis

time docker pull --platform=linux/arm/v7 hello-world
Using default tag: latest
e18f0a777aef: Download complete
f130bd2d67e6: Download complete
1ec996c686eb: Download complete
9b157615502d: Download complete
docker.io/library/hello-world:latest

________________________________________________________
Executed in    4.19 secs      fish           external
usr time   55.65 millis    0.14 millis   55.51 millis
sys time   44.06 millis    2.42 millis   41.65 millis

On 20.10 (no containerd integration, but different environment, so networking is different)

time docker pull --platform=linux/arm hello-world
Using default tag: latest
latest: Pulling from library/hello-world
9b157615502d: Pull complete
Digest: sha256:e18f0a777aefabe047a671ab3ec3eed05414477c951ab1a6f352a06974245fe7
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

real    0m2.813s
user    0m0.018s
sys 0m0.011s

time docker pull --platform=linux/arm/v7 hello-world
Using default tag: latest
latest: Pulling from library/hello-world
9b157615502d: Pull complete
Digest: sha256:e18f0a777aefabe047a671ab3ec3eed05414477c951ab1a6f352a06974245fe7
Status: Downloaded newer image for hello-world:latest
docker.io/library/hello-world:latest

real    0m2.507s
user    0m0.013s
sys 0m0.020s

moby / moby