search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.9k stars 1.31k forks source link

symlink Permission denied occurs when pull using the Crictl command #8024

Closed byeong0 closed 9 months ago

byeong0 commented 1 year ago

When using docker pull, the image is pulled normally, but when using Crictl to pull the image, the following symlink permission denied error occurs.

Expected Behavior

The docker image should pull normally.

Current Behavior

When using docker pull, the image is pulled normally, but when using Crictl to pull the image, the following symlink permission denied error occurs. 

$sudo crictl pull  docker.io/calico/node:v3.25.1
DEBU[0000] get image connection
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:docker.io/calico/node:v3.25.1,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
E0917 23:55:07.345848   21476 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"docker.io/calico/node:v3.25.1\": failed to extract layer sha256:b1d7f02a32791d579abb161bccbf82ba1deaa7fb57805c93e84ddd30f0cb9560: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3830924437: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3830924437/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown" image="docker.io/calico/node:v3.25.1"
FATA[0001] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/calico/node:v3.25.1": failed to extract layer sha256:b1d7f02a32791d579abb161bccbf82ba1deaa7fb57805c93e84ddd30f0cb9560: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3830924437: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3830924437/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown

Possible Solution

Steps to Reproduce (for bugs)

  1. sudo crictl pull docker.io/calico/node:v3.25.1

Context

When deploying calico using containerd on kubetnetes, pulling docker.io/calico/node:v3.25.1 image with symlink permisiion error and fall into CrashLoopBackOff status.

The docker.io/calico/node:v3.25.1 image should pull normally.

Your Environment

lwr20 commented 1 year ago

FWIW, we have overnight tests that run Calico in kubernetes using containerd, and we don't see this issue.

Looks like they use crictl version 1.12

Possible regression in containerd or crictl?

lwr20 commented 1 year ago

FWIW docker pulls this image just fine. I'm definitely leaning towards a crictl problem?

byeong0 commented 1 year ago

No, I don't think it's a crictl issue. I have the same problem using the nerdctl tool. https://github.com/containerd/nerdctl

$nerdctl pull --debug docker.io/calico/node:v3.25.1
DEBU[0000] verifying process skipped
DEBU[0000] Ignoring hosts dir "/etc/containerd/certs.d"  error="stat /etc/containerd/certs.d: no such file or directory"
DEBU[0000] Ignoring hosts dir "/etc/docker/certs.d"      error="stat /etc/docker/certs.d: no such file or directory"
DEBU[0000] The image will be unpacked for platform {"amd64" "linux" "" [] ""}, snapshotter "overlayfs".
DEBU[0000] fetching                                      image="docker.io/calico/node:v3.25.1"
DEBU[0000] resolving                                     host=registry-1.docker.io
DEBU[0000] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.7.6+unknown request.method=HEAD url="https://registry-1.docker.io/v2/calico/node/manifests/v3.25.1"
docker.io/calico/node:v3.25.1: resolving      |--------------------------------------|
elapsed: 0.4 s                 total:   0.0 B (0.0 B/s)
DEBU[0000] fetch response received                       host=registry-1.docker.io response.header.content-length=154 response.header.content-type=application/json response.header.date="Thu, 21 Sep 2023 08:29:32 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=202.8.174.194 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:calico/node:pull\"" response.status="401 Unauthorized" url="https://registry-1.docker.io/v2/calico/node/manifests/v3.25.1"
DEBU[0000] Unauthorized                                  header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:calico/node:pull\"" host=registry-1.docker.io
DEBU[0000] do request                                    host=registry-1.docker.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2
docker.io/calico/node:v3.25.1: resolving      |--------------------------------------|
elapsed: 1.3 s                 total:   0.0 B (0.0 B/s)
DEBU[0001] fetch response received                       host=registry-1.docker.io response.header.content-length=1697 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 21 Sep 2023 08:29:33 GMT" response.header.docker-content-digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=ca5134a1-fa66-4171-bb2f-c4c68d1d3d54 response.header.etag="\"sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f\"" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" url="https://registry-1.docker.io/v2/calico/node/manifests/v3.25.1"
DEBU[0001] resolved                                      desc.digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" host=registry-1.docker.io
DEBU[0001] fetch                                         digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1697
DEBU[0001] do request                                    digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json reques
docker.io/calico/node:v3.25.1:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f: downloading    |--------------------------------------|    0.0 B/1.7 KiB
elapsed: 1.9 s                                                                 total:   0.0 B (0.0 B/s)
DEBU[0002] fetch response received                       digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=154 response.header.content-type=application/json response.header.date="Thu, 21 Sep 2023 08:29:33 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=202.8.174.194 response.header.strict-transport-security="max-age=31536000" response.header.www-authenticate="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:calico/node:pull\"" response.status="401 Unauthorized" size=1697 url="https://registry-1.docker.io/v2/calico/node/manifests/sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f"
DEBU[0002] Unauthorized                                  digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" header="Bearer realm=\"https://auth.docker.io/token\",service=\"registry.docker.io\",scope=\"repository:calico/node:pull\"" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1697
docker.io/calico/node:v3.25.1:                                                 resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f: downloading    |--------------------------------------|    0.0 B/1.7 KiB
elapsed: 3.1 s                                                                 total:   0.0 B (0.0 B/s)
DEBU[0003] fetch response received                       digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=1697 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 21 Sep 2023 08:29:34 GMT" response.header.docker-content-digest="sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f" response.header.docker-distribution-api-version=registry/2.0 response.header.docker-ratelimit-source=ca5134a1-fa66-4171-bb2f-c4c68d1d3d54 response.header.etag="\"sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f\"" response.header.strict-transport-security="max-age=31536000" response.status="200 OK" size=1697 url="https://registry-1.docker.io/v2/calico/node/manifests/sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f"
DEBU[0003] fetch                                         digest="sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da" mediatype=application/vnd.docker.distribution.manifest.v2+json size=737
docker.io/calico/node:v3.25.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da: downloading    |--------------------------------------|    0.0 B/737.0 B
elapsed: 3.3 s                                                                    total:  1.7 Ki (514.0 B/s)
DEBU[0003] fetch response received                       digest="sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=737 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Thu, 21 Sep 2023 08:29:35 GMT" response.header.docker-content-digest="sha256:68fc6b7
docker.io/calico/node:v3.25.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da: downloading    |--------------------------------------|    0.0 B/737.0 B
docker.io/calico/node:v3.25.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:cae61b85e9b45aad28474600edd1a81f3de281917516191b160eeed4275977d2:   downloading    |--------------------------------------|    0.0 B/2.8 KiB
elapsed: 3.6 s                                                                    total:  2.4 Ki (676.0 B/s)
DEBU[0003] fetch response received                       digest="sha256:cae61b85e9b45aad28474600edd1a81f3de281917516191b160eeed4275977d2" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=262778 response.header.cache-control="public, max-age=14400" response.header.cf-cache-status=HIT response.header.cf-ray=80a0f0f8498ec0f6-ICN response.header.connection=keep-alive response.header.content-length=2825 response.header.content-type=application/octet-stream response.header.date="Thu, 21 Sep 2023 08:29:35 GMT" response.header.etag="\"460cd2febd8e222fe17c3f6e7c24d89e\"" response.header.expires="Thu, 21 Sep 2023 12:29:35 GMT" response.header.last-modified="Fri, 31 Mar 2023 01:08:42 GMT" response.header.server=cloudflare response.header.vary=Accept-Encoding response.header.x-amz-id-2="yJWmVIzqUhgoCjA4wU90H9vAiU2iwxSAraDzUZ8zRQXd6DAYv432g7H38P7IQpK3IeMdHVK+G10=" response.header.x-amz-request-id=TTGN7D0QKDC599GZ response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=knlz1ABtISQYUsDg9TBs4mxzTUKEjV_l response.status="200 OK" size=2825 url="https://registry-1.docker.io/v2/calico/node/blobs/sha256:cae61b85e9b45aad28474600edd1a81f3de281917516191b160eeed4275977d2"
DEBU[0003] fetch                                         digest="sha256:de3d34951e105833fda0ccefc8171f7bc42ff2e678eb042ece817a3c2232ed5d" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4059
DEBU[0003] fetch                                         digest="sha256:6c8ba610e03006748516517622e10428c11d069148b10734becf23f3bf8cb8f7" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=88326473
DEBU[0003] do request                                    digest="sha256:6c8ba610e03006748516517622e10428c11d069148b10734becf23f3bf8cb8f7" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header
docker.io/calico/node:v3.25.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:cae61b85e9b45aad28474600edd1a81f3de281917516191b160eeed4275977d2:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:de3d34951e105833fda0ccefc8171f7bc42ff2e678eb042ece817a3c2232ed5d:    downloading    |--------------------------------------|    0.0 B/4.0 KiB
layer-sha256:6c8ba610e03006748516517622e10428c11d069148b10734becf23f3bf8cb8f7:    downloading    |--------------------------------------|    0.0 B/84.2 MiB
docker.io/calico/node:v3.25.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:cae61b85e9b45aad28474600edd1a81f3de281917516191b160eeed4275977d2:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:de3d34951e105833fda0ccefc8171f7bc42ff2e678eb042ece817a3c2232ed5d:    downloading    |--------------------------------------|    0.0 B/4.0 KiB
layer-sha256:6c8ba610e03006748516517622e10428c11d069148b10734becf23f3bf8cb8f7:    downloading    |++------------------------------------|  5.0 MiB/84.2 MiB
docker.io/calico/node:v3.25.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:0cd00e83d06b3af8cd712ad2c310be07b240235ad7ca1397e04eb14d20dcc20f:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:68fc6b7a097fab48a442e4572ccb0d3957665ade2a55a65631256500576d89da: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:cae61b85e9b45aad28474600edd1a81f3de281917516191b160eeed4275977d2:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:de3d34951e105833fda0ccefc8171f7bc42ff2e678eb042ece817a3c2232ed5d:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6c8ba610e03006748516517622e10428c11d069148b10734becf23f3bf8cb8f7:    downloading    |+++++++++++++++++++++++++++++++++++---| 79.0 MiB/84.2 MiB
elapsed: 4.9 s                                                                    total:  79.0 M (16.1 MiB/s)
FATA[0005] failed to extract layer sha256:b1d7f02a32791d579abb161bccbf82ba1deaa7fb57805c93e84ddd30f0cb9560: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3733508295: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3733508295/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown

could it be an issue with containerd?

lwr20 commented 1 year ago

could it be an issue with containerd?

Maybe? I think there are a few possibilities, including:

  1. docker (which we use to build this image) isn't producing an OCI compliant image
  2. containerd (which you're using to load this image) isn't loading an OCI compliant image
  3. the image has been corrupted in its journey (seems unlikely, given that its hash is checked at various points)
  4. user error of some sort where pulling this image needs a special flag/permissions/user to work.

but I'm not sure the Calico team are in a position to fix any of these? Perhaps raise with the containerd folks and see what they say? I think they will understand this problem better than I do, anyway :)

byeong0 commented 11 months ago

Some versions of calico/node images can be pulled normally.

$ sudo ./nerdctl pull docker.io/calico/node:v3.23.4
docker.io/calico/node:v3.23.4:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:001daf5e8d6b9b3cba53a585ce913222799f66c3912443861878c5dbeb17aeaa:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:3e213ba8da770f1567590ca87b4c3dfb0c5247962f773ad5b6c1004a5cb91c11: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:7c163e840d371e80f31b0d2096f4dec1b284d7b479630f14ae44e929adc26b8a:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:85b8bb2d8c8ed06f5b97b55fbfda41598bd9125b503d4ad9584032ed439eb945:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d27541c40274faf58552babfe10e1bc7d357c6dcef7dedf1cf1fbf97c45934d5:    downloading    |+++++++++++++++++++++++++++++++++-----| 64.0 MiB/71.6 MiB
elapsed: 6.6 s                                                                    total:  64.0 M (9.7 MiB/s)
FATA[0006] failed to extract layer sha256:1f2b060a7a5354119d14274b76f47b24321357c547253d9ae4cfcf3910db676e: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3785089323: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3785089323/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.23.3
docker.io/calico/node:v3.23.3:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:b356c2334729810de4781819ac7cf7cb05e49b8be9387e6bba2755df95d1cd84:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:4ceac15651be3e8ed411266d2941f8ff04f0efbef9b43149f8b231664725a1e8: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:5f5175f39b19e9aa0a69889295778b6a8ae8b2858bb691fd89a3941b949a6fac:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e7590a44ed0d7284bc76d3b93b0870af4ee28405c6a195521b51362afd6ac125:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9bf5eb736b3c10577803091d0717973411c2e25cf7c9588da969fe94a4ffc516:    downloading    |++++++++++++++++++++++++++++++--------| 56.0 MiB/70.0 MiB
elapsed: 4.5 s                                                                    total:  56.0 M (12.4 MiB/s)
FATA[0004] failed to extract layer sha256:87d23e04f6cc18f43788b914cf2d0ad142edb701c63e1959bf9a12dc0dbb3726: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3018255422: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3018255422/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.23.2
docker.io/calico/node:v3.23.2:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:b4ac0660c297b3a582ef2f4a0d7ef86f954ad5497b704b41d82fa99418e7a51e:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0a430186f9267218aed3e419a541e306eba3d0bbe5cf4f6a8b700d35d7a4f7fa: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:a3447b26d32c74054202a7a312bc78fe4257c157272048efd566fdb321ecf1be:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:2a9798995149ad3a229b7deab1eb1624af8fbc0bac7e2667566f0d2ed60f0d5d:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:99bf505ac835fb2efbc7734c2e76f53f8ad67ae4b301052382661c774bcfc8e3:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 7.3 s                                                                    total:  74.2 M (10.2 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/node:v3.23.1
docker.io/calico/node:v3.23.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:d2c1613ef26c9ad43af40527691db1f3ad640291d5e4655ae27f1dd9222cc380:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:9a59f19f6205a45fc34d33e3bd0044ce2f552a45d18a83af75e83402a84403f1: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:fbfd04bbb7f471a173dd322b9240e20c65ec450efc9030221d44e703c38add0e:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5f39b8e1c11f63ce18d82e5bcd4d78e0962201c97e926fd67ef3497692b744af:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b62ec490a6d4e210b85ef411dc7628d8c8523297df42f55ae55e0b5d44a68218:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 8.9 s                                                                    total:  73.0 M (8.2 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/node:v3.22.1
docker.io/calico/node:v3.22.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:1f8ed83e5264b4206cce7e1def11bca0b3ea7d5f4eb9b0ca0dbfc8cb968ca57e:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:2d997ad1452537cb06c76d6a92a4cce3bad699737cb28ada371ad0c1ae824e4a: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:7a71aca7b60fcb933d5f8e30e17fde66c4d1ff8a963bfde1ee0c43d5a4ed577d:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4ee411b1f5a98f9af1acee18ea9e441a0753d30af485797e785b3297c7b0d17c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:84aae6a8ca4741c7a81b199f61e53cfa3e176f8f034270df1b02a6e4e1089579:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 6.8 s                                                                    total:  66.4 M (9.8 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/node:v3.22.2
docker.io/calico/node:v3.22.2:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:41aac6d0a440746f03d03a2912d92809681d07b6ce0ca4649241a1b95ff5e513:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6c78357866f91785c02ce67a0432aefc1d1031c61a60f2641ab70398469a5bc9: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:fd1608dbbc197635485022a89e66ba0a20386ca400cf67252c6fde22d3335d9f:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8796bae27287fb9737623967c17f4f6de9c8e69c4ebe3610a98c01114b802acb:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5d8943abbb96629026d37a1d058e32cee203d8b18846853de1c7dcd91c10c4f2:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 7.3 s                                                                    total:  66.5 M (9.1 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/node:v3.22.3
docker.io/calico/node:v3.22.3:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:c3761112198df42ae052488656d37933a8f79bdb2c075ff77d5c499afaf12602:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:27ebbf12234943575854a3eeb6b499e19576ed8fa1843dc416adb944440fc5cb: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:6d43d7bb6dffd810ca3645bb4ddd5173c626bf212567ededb44282a37c2bd49f:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a9997de50cd59ebd1040cba780aa3849f5ad187003b8ae89db17f6b28a266ec3:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:289e97ef16f3848d09162022f423f095ffcaee864f5a5844f8f55f52b08234fe:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 8.5 s                                                                    total:  71.3 M (8.4 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/node:v3.22.4
docker.io/calico/node:v3.22.4:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:31d876566be2ed2fdcf5f6e1af698e7b2191ff721230ec7e858b61c409bc08e7:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:e8d9fc21ac7cf51744e0747defa85d05aa7e4ffb2324d177bba10a41b111b36c: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:3a6768785333dfa290e31d8c7ad99874f0f97a3319f7b424000fa2671cd007af:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3b99905040069a4ed0e13dc0db3564c8504f9246493f0ed7a759d8ba4f6a3136:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a340b1e8d0af5f50cbc14ff1fde7a9ea1a64b8d32dd6b5d533e3b481862e56d9:    downloading    |++++++++++++++++++++++++++++++++++----| 62.0 MiB/68.1 MiB
elapsed: 6.4 s                                                                    total:  62.0 M (9.7 MiB/s)
FATA[0006] failed to extract layer sha256:4976be3f9c6fb86aa65f1c63a13e894d24ebc9606b7e00bbec94243ea5401a5b: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3078571876: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3078571876/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.22.5
docker.io/calico/node:v3.22.5:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:a53e7849aea34dd832fd01c034fdf93ec39ea007e11e5a2fc62dd79416d293cb:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:83262189f1396729526fd4a70c12131c4516d1f732a2cfb299b7264cd9b51a89: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:30308f97f742f73d2f4405ddb31182bcb7a9f9f093fa6b382b33f97697b540ce:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:484faf4588dd7d0f015114b5d00c9c0cbdce8a6cb1a46d66a8470873b18e7f82:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0483b33e26dd92d741ea078e2f68ad4396a81e14198bec027b75623a5b41916a:    downloading    |++++++++++++++++++++++++++++++++------| 60.0 MiB/69.7 MiB
elapsed: 4.6 s                                                                    total:  60.0 M (13.0 MiB/s)
FATA[0004] failed to extract layer sha256:595bfe53403bf3cf7c7915db7e5a6ca3e09699a65a6f2d82c8228ed0c479f6a2: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1228517757: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount1228517757/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.24.1
[sudo] password for scom:
docker.io/calico/node:v3.24.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:43f6cee5ca002505ea142b3821a76d585aa0c8d22bc58b7e48589ca7deb48c13:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:135054e0bc904c308a835689743ae00170dbb5b5dad36ba8c498c6af5799ea14: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:75392e3500e3675026eb95f4b400e9af90a9a48616ec6b5ed93883a9fb60f7dc:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:caa813daed01d2f8ecb43210fe733c3a0eaffc96ba81f27de5f9ee50c9cdcc20:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3e9e0b5458ddf389525fdb171c058fa3a83a8c1e0e89572c4b3d5f0e054e33b6:    downloading    |+++++++++++++++++++++++++++++++++-----| 68.0 MiB/76.5 MiB
elapsed: 5.3 s                                                                    total:  68.0 M (12.8 MiB/s)
FATA[0005] failed to extract layer sha256:f0b4644b09acc454e1e5f4af6bfcac2092590fd5084191aa982e0610c7d1980e: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount2881991573: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount2881991573/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.24.2
docker.io/calico/node:v3.24.2:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:84197680f76ce6bd341c077d8a4a4bb7a32c7eb2a75130f8fafc72e289d3b855:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:6c3b93a21254661458f8db46573c28d0bf2d25503f4ff2d0bce60f534d78486e: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:a7de69da7d13a218b18ca8252673a6c274f607680a3560275e6e814e047d1208:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:42cc2b82ff5275b80a5ea614bad4d292dbbbc8bd5e0fc1e337d329b1dc8644bb:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6e186ea57594ae860af0ff11bdfe8c5d4860832588a37eebaafb333a045eb420:    downloading    |++++++++++++++++++++++++++++++++++----| 71.0 MiB/77.6 MiB
elapsed: 6.8 s                                                                    total:  71.0 M (10.4 MiB/s)
FATA[0006] failed to extract layer sha256:2911a1b79977e6dd774eb88753c1a010a39ae53494b037576ca526c3f92ef42c: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount1608117675: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount1608117675/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.24.3
docker.io/calico/node:v3.24.3:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:aadbce6b3fd04d11b367e28225e5382d45c0c9c998f62dafb9b5bf1e2b231ec7:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:5b611bd32025dde645fcedc40d0764707d68146571561b0285cdd6fd3b2762c9: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:d1350774fea6cc289dd1855a8d0f97e8cdaffd79e272fff0fbaab45defd75a03:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7e108f5e2e212f582999560267e3d4929026b9a84e894289faa2af687fd77b23:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1ecaf1a71248bb31d24057b7895cee3b9b362739750042ef357f30de23d3ddc7:    downloading    |+++++++++++++++++++++++++++++++++++---| 73.0 MiB/77.8 MiB
elapsed: 6.8 s                                                                    total:  73.0 M (10.7 MiB/s)
FATA[0006] failed to extract layer sha256:fdc1b97df13f0627732ecae5edc1c0520baa69da308f15bc0676094ea8778c3a: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount103843857: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount103843857/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/node:v3.24.4
docker.io/calico/node:v3.24.4:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:68dd802f1f6d1c9e564950d72757b275bb8755bced64faaaa004b2979f8f38bc:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:f92609abe33e5b432222e44443a66089c5c4ecd779b9c2456515b1e0129ff63f: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:32c45127e587f06a38be35959ffa327e90ea6f83a11add5c25c84f190f0ee2ea:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:1b9eea0aab45ad9f5b1991e5661ac1c12f8afb9a2a023799bf319793fc6ee0a9:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:42055bcf4866865ed750e1b9e38304ed7f7a5a25aec2ef3f7244e3e8b233d4e9:    downloading    |++++++++++++++++++++++++++++++++------| 66.0 MiB/77.3 MiB
elapsed: 6.8 s                                                                    total:  66.0 M (9.7 MiB/s)
FATA[0006] failed to extract layer sha256:330a4044ba0f6705c5c68215dbc091fc68de24eb02d4360d05193e31f56569f3: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount3485981260: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount3485981260/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown

Anything other than calico/node images will be pulled normally.

$ sudo ./nerdctl pull docker.io/calico/typha/v3.26.1
docker.io/calico/typha/v3.26.1:latest: resolving      |--------------------------------------|
elapsed: 1.2 s                         total:   0.0 B (0.0 B/s)
docker.io/calico/typha:v3.26.1:                                                   resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:54ec33e1e6a6f2a7694b29e7101934ba75752e0c4543ee988a4015ce0caa7b99:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:7df19465017798019c8b2c0137b9a12ea288373d07b6d78513b4ac6e84513cbc: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:66bcdacc0ea35225cfcd841af791ceab41990622f8beef1778433b4a33000336:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:d5a5d07de83a6038dbef2886de14c6189deab7dacea17f93b6829790b04ec277:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7d0ebc32d30280fba1702ec3d248917941e2ddad3d20e490e8d69d6656df5029:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:69f331ffbc69f1ce2251af4ad1edaa89bacc8d46a5fed5deda09992c375effd4:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:129bdae6dfdfb45c01de9df23aa38f87514600e08339b69eb8255fc83314f1a7:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:686fe115d7d4c7472780674e388669d0964462e68df881a3d94a26f227de9a8c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5f7ca932a374cc17e8788ebcd14f11546c911cd6cb9646077aa3286a7661a472:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fc0a7363d5050ec6457466c9669a7e1e3f6efe7a2ac8d554ea724d2e45537130:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8577084e8adb796ac12807731fcc0806a4891d9e9132e1b40eb9c5dc20e8c0a2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c9a6458b8ce133261011e19932d788f9074e7eb16ef9ceebf8d7206fd04cf2a4:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9b70ba75ceeadf4bcfd6972721178692de2777c863fc09388d9b7c53297e90a5:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:768d82daff38c70fb8fddea12cade744b4e44b4ff92428755de952077f125863:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bc79b940c4165514d9c4150ce000f55857734957b5d47255e73d5811b9c41c42:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8280450b13dd14455eda189b4f56ec3e43d1cf5b02052eea3e048082bca9859e:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.4 s                                                                    total:  27.0 M (5.0 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/ctl:v3.26.1
docker.io/calico/ctl:v3.26.1:                                                     resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:c8e10f3c8075085fdd79156d927fd6493868d854d9425044b6083b0b53e348a0:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:0cbb127bcaf7a330c5b3163c901db50427f0879c3dacdb2abf069ab189765d0f: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:899b53c8eb001dd01e4dda71604c1476cb1141a59aa967f92e09700309d99ebe:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bfdb17f2fd34153a1fb49a71a076bccdeb9988378271ace2091c7e1d66b06fb6:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:363d83e78aeb2fc0e341357096e78d5c5a65e21ca2c05c2d05e66eeed522fcc3:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7b819f1211a41c09087cdf8aa666adff8b644ddc5370560b7f2f08cbc11af8f2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5be9846711f8a84100369a529e5aea39b0751442883f4d66b75320ee51088186:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4def3dea0949b545027bb5bad201693132547c0050e46bf3191128570428aaa7:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.7 s                                                                    total:  26.1 M (4.6 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/node:v3.26.1
docker.io/calico/node:v3.26.1:                                                    resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:8e34517775f319917a0be516ed3a373dbfca650d1ee8e72158087c24356f47fb:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:9459d1b2831955120fdf0037e6816b21e5d88dd11110d6d89398e5ef53cdf54c: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:8065b798a4d6729605e3706c202db657bfbcb8109127ece6af5bfb6da106adb7:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6cf92fa5251d7e452a867aeff683770a48ccedb470a895c3c4586ccfc54b998b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c998069b4a7c3648fbb90895688851dc8c08491fc9dbbfcafdb0a020499ce33b:    downloading    |+++++++++++++++++++++++++++++++++-----| 73.0 MiB/82.6 MiB
elapsed: 4.9 s                                                                    total:  73.0 M (14.9 MiB/s)
FATA[0005] failed to extract layer sha256:5ed6bb5a36424feb61d1a2e9fa1805e9bf3be47453e9700d5d46e623b68ed663: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount2852629097: symlink /usr/lib/systemd/system/reboot.target /var/lib/containerd/tmpmounts/containerd-mount2852629097/etc/systemd/system/ctrl-alt-del.target: permission denied: unknown
$ sudo ./nerdctl pull docker.io/calico/cni:v3.26.1
docker.io/calico/cni:v3.26.1:                                                     resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:3be3c67ddba17004c292eafec98cc49368ac273b40b27c8a6621be4471d348d6:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:86779fab56f3c0c51abcae6d5c5d712f54ed86b50eebf83e54b8c80fdcb4a76e: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:9dee260ef7f5990aaf6e8f6767b767366c27a6abbf79ba8dba45ff3290bd5de0:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ead1d93126e20f9f5ad880901c1bdbc81a39c94866279cddcee478428292e593:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3d817138ad224b687ef69f9a7eadae6e8cb4cb39063d72c455a75953211aa400:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:263e6c47f9f4e68dd7262ff6da1756236b66ff2ed7c766d04a6eafe69e9ca1d7:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:85790a7a5aee466e779d1d8c7d49ad74a3cefe752ad3b60813f54ddbdffe6b32:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:cec6d7c8ed45533ae61509dd5b996231c3a4d9e91375bf31141535c151dd5abc:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:93a53306e2e3dbee87f4ff4b6df9cb5a21fe5c6a9dbfb105a0c7a177d288118b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a0ff5af14493c454aea61d8a2f644d258100643842e2d733228abd3a1a0ba593:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:81ce5fcda971481751c133e27d087909e4259157be9e32a4c96a393e7fc6eb78:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a13d926928137cdf44c4711e5bcfd2ac1b8f9f1875f5d04c4d247746441a2410:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:31f2a5683ad7183a3d38ab46acd18703cbcb462605ea845d4a03ede6807ecfe6:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:32305daf67720b47d69565e015cebc8a718b5f2397f2c83f8ad5ef72a20ae2c7:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 3.3 s                                                                    total:  89.0 M (27.0 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/apiserver:v3.26.1
docker.io/calico/apiserver:v3.26.1:                                               resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:8040d37d7038ec4a24b5d670526c7a1da432f825e6aad1cb4ea1fc24cb13fc1e:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:5c5b60cf0141ee28df5a9194d67b4fc6d59f6adfdc145d5d21044137d4df7ab0: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:c4a7294544c6bee1a71522ef0c720b85c5aab6cdc533cca9a059360f842fb9b2:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ebdea64c42b570a4cb6056d059d8a7d7e382d5d31b1dfa90c33beebb36500314:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9c3790ff827546337b686abb4114956b80680af6fc9531e114e2a602f95d010e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    exists         |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3cd76a54f800ba60cf47bd24b017ee10d8e334f409216ff9b748ac202f251a27:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f7e5e13ac19025e63d2a9af2a9dada618ddd3bee46dc0220b95e4d066c2b20ee:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:15d5f1149c947ac5089184be55eb528a6e5389436145c865d5597a2853dd52db:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:a8563debd54bbc04418c70739f8eea00d62c63b963ccfdc339c203a411cc9e2b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f825632f7d6a064e109665fc78bc8af0ca4abe9a3794f371a59a5a804b23997c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e196ec54787fdb59b2809834309da2d0a4c5f6e724142537ffbdb6f9129f5ddc:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:645eabbea4a0f4e33f6a597327335a0bd44a895c1c1602562f60cdcd25ff8216:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:7f7abcf0eec912c214baf57432bf7216687d8c06171154d3c379c2707f194838:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4903ad77c531cf7fa554044dbdea7d1bd8ad95d23b47eab48b39198198c4ab4b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e5451de8cf77c00860391709bb42a62ae555788e1148ce00f3c58f376aae6620:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.6 s                                                                    total:  37.7 M (6.7 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/kube-controllers:v3.26.1
docker.io/calico/kube-controllers:v3.26.1:                                        resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:01ce29ea8f2b34b6cef904f526baed98db4c0581102f194e36f2cd97943f77aa:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:2c5526ad8cd69740448207b90f4077fd68a5d2e922014e32141b38a529295c55: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:1919f2787fa7098a4c0dec514cf385f90a79c9f6c5effecda570fa3b8c771a3d:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:fe2ccd270eb72080c820d6d06c7ecdc1cbb635437da0a73087ee89a8fd2917c9:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:8152bb18c4b9585e54b5e09144bd706888fbf7186fd363849c0d2909217177d6:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5fcafcc9e60eba36a3c9bdb19a03a8da6d7250660671108cac301ddace02cf8a:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c383daff5dcb5ffc2aa48cadeff8e353d1c072112447e0317650826674546cdd:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b9b2f2cb965a90c288495a957456ca400ea7495313fbcd86f4f048b68c2a5d84:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5cd0c04b752552887129663a2cde0568af25423a62d008bc545e0579797fa5b8:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:566290fdc43c5d9ec5ca4c0f74b9bd7d3935252f46d3aaedb4e9548cf690eb9f:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:58bd207ae1113730cdd4f0f9d09d1d052e06f40373b196b0c32a590d5202fff0:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:807b54c4290109ef9a081d6f267efa69186b7b6941bed4efdcdff167bdd394a2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:96510a036d14ee0218bbf21514ffbc7178cc2f2d69edcf9d9eb7296f694dd20c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:26bd6f1dcca4edc212f305d253227a36c98f399f12a1eb0ce7955eca68314d24:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:6e777fc9ae2747d305defb5bfcde5bc544451df798082887dc2e293662100ef7:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:70e5fa918535de095bc2c0d955c1434f8b7c0466812babc789ae94cd93b5e8c2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:94d15eeac54c61bdbd1c0da5ca031095144aac70399e67b2d240738c5f99043c:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 5.4 s                                                                    total:  31.3 M (5.8 MiB/s)
$ sudo ./nerdctl pull docker.io/calico/flannel-migration-controller:v3.26.1
docker.io/calico/flannel-migration-controller:v3.26.1:                            resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:ee3774e5e8672a4b809a0ff1c229c25e0129851f21099639bac778e6d82f98c3:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:16a12c27276c2ae7d11b78b5bd7dad8306885a5594e6d1aff99e9e598af85abf: done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:6981b395ac9479fb928ff493d21197f77c2b51d3e0cbfecc80387e8c9d076553:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0bbc6f63eac3fe64a14ee3a59b112789edde2eefd10e22825ab3da20d842dd7e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:ec8bf22cb33ed65f7ffacee6f790242c40f46b991191922812a20d9337ffc39c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0e437ba3dca83dc4b76be8d5f218524e78c7696490d60502b965d30997035158:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:c684004ef02513dd711b6ed9ff8164d1c9de217383c83cdd486a778570557e7b:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f90d1bb2d9774e8c5c00441a29eec3afd7c8d6745006f1289e23eb8c52de8cc2:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:18ef05bf57cfa1fcde1de1222169490a788808747f9fcd5a89845c9ff0dd5da4:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bfd35eb7d003e1bc531af28551bfe8476e240b378be352ee24abab8505ef0b79:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0ef9f9623760351608b8be832b957e257fc20d8aa6670758e1f94f8f368529f1:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:df9fe522f2469631797875e546e92b160ebd67109cfd5195707c32f91fa31add:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f975b56a0374940330aea28a325544f868722d3a04f762fc7b21ba235447dd6e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e4e66af753c29fa4bad6f229919b22f9d4a403652056ef0c729f949035eb5790:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:5c4a7b7ce0685350667f92c44b70f6ae942d1e434b4becdad02fdca82d7076bb:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:f6a9c0e3846ba8d1b20417ca75d039e515fea3a0343ee7b18546d54b7e7670d8:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 7.8 s                                                                    total:  44.9 M (5.8 MiB/s)

And centos7.8.2003 (3.10.0-1127.10.1.el7.x86_64), the pull works fine.

lwr20 commented 11 months ago

And centos7.8.2003 (3.10.0-1127.10.1.el7.x86_64), the pull works fine.

Are you saying that all the failing pulls work OK on Centos 7.8.2003 (3.10.0-1127.10.1.el7.x86_64)?

lwr20 commented 11 months ago

Pinging @danudey @mgleung - any ideas here?

lwr20 commented 11 months ago 
$ sudo ./nerdctl pull docker.io/calico/node:v3.23.4
docker.io/calico/node:v3.23.4:                                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:001daf5e8d6b9b3cba53a585ce913222799f66c3912443861878c5dbeb17aeaa:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:3e213ba8da770f1567590ca87b4c3dfb0c5247962f773ad5b6c1004a5cb91c11: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:7c163e840d371e80f31b0d2096f4dec1b284d7b479630f14ae44e929adc26b8a:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:85b8bb2d8c8ed06f5b97b55fbfda41598bd9125b503d4ad9584032ed439eb945:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:d27541c40274faf58552babfe10e1bc7d357c6dcef7dedf1cf1fbf97c45934d5:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 49.8s                                                                    total:  71.6 M (1.4 MiB/s)                                       
$ echo $?
0

$ sudo ./nerdctl pull docker.io/calico/node:v3.24.4
docker.io/calico/node:v3.24.4:                                                    resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:68dd802f1f6d1c9e564950d72757b275bb8755bced64faaaa004b2979f8f38bc:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:f92609abe33e5b432222e44443a66089c5c4ecd779b9c2456515b1e0129ff63f: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:32c45127e587f06a38be35959ffa327e90ea6f83a11add5c25c84f190f0ee2ea:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:1b9eea0aab45ad9f5b1991e5661ac1c12f8afb9a2a023799bf319793fc6ee0a9:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:42055bcf4866865ed750e1b9e38304ed7f7a5a25aec2ef3f7244e3e8b233d4e9:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 54.6s                                                                    total:  77.3 M (1.4 MiB/s)                                       
$ echo $?
0

$ ./nerdctl --version
nerdctl version 1.6.0

I'm afraid it appears I can't repro the issue on my machine (Ubuntu 20.04.6 LTS)

byeong0 commented 11 months ago

And centos7.8.2003 (3.10.0-1127.10.1.el7.x86_64), the pull works fine.

Are you saying that all the failing pulls work OK on Centos 7.8.2003 (3.10.0-1127.10.1.el7.x86_64)?

Yes, All images are pulled normally on Centos 7.8.2003.

lwr20 commented 11 months ago

Yes, All images are pulled normally on Centos 7.8.2003.

That would suggest its a problem with your machine/distro then, no? I don't think there's anything the Calico team can do to help here, is there?