Containerd Image pull fails intermittently with EOF error for Harbor private registry

[charugarg2810]

Description

Hi Team,

We have recently upgraded our EKS cluster from v1.23 to 1.24, which replaces the Docker with Containerd. Now in 1.24 clusters, we are seeing imagepullbackoff issues for harbor images. While trying to pull the image manually it gives below output.

Steps to reproduce the issue

1. 2. 3.

Describe the results you received and expected

ctr -n k8s.io image pull harbor.registry.dns/image/path:1234 --user user1

Password: harbor.registry.dns: resolved |++++++++++++++++++++++++++++++++++++++| index-sha256:371b1e63285f74ff6927becf34a47f302e954c7835987c0660c63b8ac5bb4651: waiting |--------------------------------------| manifest-sha256:1ec1ffc82b90a3e9cb7445c2079cb60a954b41303b0ac9174144ded2f4cccc05: done |++++++++++++++++++++++++++++++++++++++| config-sha256:d437a0ad48fceb1463e1c2af0b35959ab5cbab3a4961009d1e964571d7d9e161: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:f868ac632a6f7994c589ce631afb99014698a783e173144bf4740e318303e992: downloading |++++++++++++++++++++++++++++++++------| 214.0 Mi/250.5 MiB layer-sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:0b172db2480d88fcc4bec429da2777b56687dbcd82d80c19dc41e16ffc56d14b: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:76bf1f85fdf38a127d6047c9b3d96651797e2cb5825172dee23cb611f3111f9c: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:fe906bbc5faed6202ead1ec035b9467f3cf02bf5120145dec708061c5e81c119: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:39182aedab0a7dad5dbd70262fbc13e54f2b2cab4f278078650936c2e7dae52a: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:381187300573beeff328abb31687c357112e575c9ca02b256683f98224cc7e93: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:52c3e4aa8e82ec89a0cedcea37e8d9b4541b00d560d85874cb271fb80288e75b: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa: exists |++++++++++++++++++++++++++++++++++++++| layer-sha256:e25e5e21d0a6a269d82a220ce0848c2862694dbee4956386a06f5e71ad8f892c: done |++++++++++++++++++++++++++++++++++++++| layer-sha256:16c640aa63795ffac2c65bdb68e5d81cbf3b640862f710151f1b0a8e4d570753: done |++++++++++++++++++++++++++++++++++++++| elapsed: 2.4 s total: 230.5 (96.0 MiB/s) ctr: failed to read expected number of bytes: unexpected EOF

What version of containerd are you using?

1.6.19

Any other relevant information

runc --version

runc version 1.1.5
commit: f19387a6bec4944c770f7668ab51c4348d9c2f38
spec: 1.0.2-dev
go: go1.18.6
libseccomp: 2.4.1

uname -a 5.10.179-171.711.amzn2.aarch64 #1 SMP Tue Jun 6 01:59:12 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux

Show configuration if it is related to CRI plugin.

No response

[fuweid]

ctr: failed to read expected number of bytes: unexpected EOF

I think it might be related to your registry. Would you mind to use debug mode to rerun it again? thanks

[charugarg2810]

# ctr --debug -n k8s.io image pull $image --user user1 harbor.registry.dns:latest sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f Password: DEBU[0001] fetching image="harbor.registry.dns/project1:198-307d1e4" DEBU[0001] resolving host=harbor.registry.dns DEBU[0001] do request host=harbor.registry.dnsrequest.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=HEAD url="https://harbor.registry.dns/v2/project1/manifests/198-307d1e4" DEBU[0001] fetch response received host=harbor.registry.dns response.header.connection=keep-alive response.header.content-length=743 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 25 Jul 2023 12:34:47 GMT" response.header.docker-content-digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=01ebd46f-8fd7-488b-9e63-4dcdd3933d95 response.status="200 OK" url="https://harbor.registry.dns/v2/project1/manifests/198-307d1e4" DEBU[0001] resolved desc.digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" host=harbor.registry.dns DEBU[0001] fetch digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=743 DEBU[0001] do request digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=743 url="https://harbor.registry.dns/v2/project1/manifests/sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" DEBU[0001] fetch response received digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.connection=keep-alive response.header.content-length=743 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Tue, 25 Jul 2023 12:34:47 GMT" response.header.docker-content-digest="sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=213557ed-5882-4eeb-861c-9f4c3dee00f1 response.status="200 OK" size=743 url="https://harbor.registry.dns/v2/project1/manifests/sha256:03d30a5448b280674d977c1c0108ca0a04176cb559cc4fd95374b4faeb059c57" DEBU[0001] fetch digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" mediatype=application/vnd.docker.distribution.manifest.v2+json size=4510 DEBU[0001] do request digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=4510 url="https://harbor.registry.dns/v2/project1/manifests/sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" DEBU[0001] fetch response received digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.connection=keep-alive response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.vary=Accept-Encoding response.header.x-request-id=9e10eb41-ed01-44fa-b6e9-6c2ed402ef32 response.status="200 OK" size=4510 url="https://harbor.registry.dns/v2/project1/manifests/sha256:b5f993a75a9ef71894f446d0805e132786b8b7a4e873c9b25fa985b96dfae8e0" DEBU[0001] fetch digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=13666138 DEBU[0001] fetch digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=161807505 DEBU[0001] fetch digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=260 DEBU[0001] fetch digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5400162 DEBU[0001] fetch digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4194546 DEBU[0001] fetch digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5869988 DEBU[0001] fetch digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1413 DEBU[0001] fetch digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40917255 DEBU[0001] fetch digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" mediatype=application/vnd.docker.container.image.v1+json size=8424 DEBU[0001] fetch digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11517 DEBU[0001] fetch digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=921635 DEBU[0001] fetch digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=156125588 DEBU[0001] fetch digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5108 DEBU[0001] fetch digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6551 DEBU[0001] fetch digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4561395 DEBU[0001] fetch digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1001 DEBU[0001] fetch digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=980 DEBU[0001] fetch digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=292 DEBU[0001] fetch digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=32 DEBU[0001] fetch digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5118 DEBU[0001] fetch digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=148608462 DEBU[0001] do request digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=13666138 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" DEBU[0001] do request digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=161807505 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" DEBU[0001] do request digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=260 url="https://harbor.registry.dns/v2/project1/blobs/sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" DEBU[0001] fetch response received digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=13666138 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=930504fd-e1ed-46e1-9fce-ce5edf55f2cf response.status="200 OK" size=13666138 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6fb9d4937c2de8d137362d704c8f8de084db756c885c5474d20dd69044024998" DEBU[0001] do request digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=4194546 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" DEBU[0001] do request digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5400162 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" DEBU[0001] do request digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=921635 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" DEBU[0001] do request digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5869988 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" DEBU[0001] do request digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=1413 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" DEBU[0001] fetch response received digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=260 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=2830f2e7-6952-4a29-864c-c7a89eb8c41f response.status="200 OK" size=260 url="https://harbor.registry.dns/v2/project1/blobs/sha256:f4ca923fee1c63e06eb59c3c36ee6d9f2c5109c36f65faa6ef1438665af7d2c8" DEBU[0001] do request digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=156125588 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" DEBU[0001] do request digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=40917255 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" DEBU[0001] fetch response received digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=161807505 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=fb8f6a07-74c8-4118-856a-72fcf7b5ec9c response.status="200 OK" size=161807505 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7bbbfc19b082760596a0f4597bd4690f205a855d1200a27a3f9a8a402dce5a23" DEBU[0001] fetch response received digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=921635 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=a7014d4e-7a52-4d06-8b35-c2de02edede8 response.status="200 OK"size=921635 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b042aaf118281415531e49b79872d7acaccc799fc45c155968386191d5b0fe96" DEBU[0001] fetch response received digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=4194546 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=5f60a1dc-2250-4b9c-b0cb-0065276429e1 response.status="200 OK" size=4194546 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d1978f9b30d3ea4c690f3b1d639eb3040c3d56a1dbf89363fa977585f7c202ea" DEBU[0001] do request digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5108 url="https://harbor.registry.dns/v2/project1/blobs/sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" DEBU[0001] do request digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=8424 url="https://harbor.registry.dns/v2/project1/blobs/sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" DEBU[0001] fetch response received digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5400162 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=cafddd3e-edef-4a28-955a-b74703c6ac27 response.status="200 OK" size=5400162 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c24ba2ae0a6163df9a3d5fe87a113ae6fac7daab71ff962b62a3558bd58442e" DEBU[0001] fetch response received digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5869988 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=990996c1-085f-4458-b04e-47cdd9f49e02 response.status="200 OK" size=5869988 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6bd3d27bfa3c0fe6792495d572d0a948e6655fb9aacaa1620fe586d5b9b7cd3f" DEBU[0001] fetch response received digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=1413 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=5d030e7d-e717-4c1d-b172-c89ca1ab74db response.status="200 OK" size=1413 url="https://harbor.registry.dns/v2/project1/blobs/sha256:b7d1e8b898d55958e442914f4ab2da7caffb60282e4025d457adecc74ae27a92" DEBU[0001] fetch response received digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=40917255 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=409986c6-43c6-4984-9454-c9114a894f6a response.status="200 OK" size=40917255 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2c594df67f226b623622ccac83bf2cea5024eab532cf930fc6f888011906a1fa" DEBU[0001] do request digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=6551 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" DEBU[0001] do request digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=11517 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" DEBU[0001] fetch response received digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5108 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=d3d8c186-f8d5-4dbd-bce6-ba1d8a7beebe response.status="200 OK" size=5108 url="https://harbor.registry.dns/v2/project1/blobs/sha256:be84b89f0a92df4ab5bbad3df19cbb511ebce4b419043fa96ffa5d51fb8eb2a3" DEBU[0001] fetch response received digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=156125588 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=0a4414e4-0523-4cd4-ad9b-3a0ce18201b8 response.status="200 OK" size=156125588 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" DEBU[0001] fetch response received digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=8424 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=fdf14afd-0326-4e8d-89a1-915a1ed69f14 response.status="200 OK" size=8424 url="https://harbor.registry.dns/v2/project1/blobs/sha256:61f78f237d3723e4c98cf685bd1fbf0001dc1d2ae5cd36a1848c062f67ae1a0f" DEBU[0001] do request digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=4561395 url="https://harbor.registry.dns/v2/project1/blobs/sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" DEBU[0001] fetch response received digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=6551 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=cd5e2e65-2781-4e51-9e94-90aacb0e5864 response.status="200 OK" size=6551 url="https://harbor.registry.dns/v2/project1/blobs/sha256:6c9601c677052f34717209df96af5cb97a65f46af42afa86f69c946af48e0348" DEBU[0001] do request digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=980 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" DEBU[0001] fetch response received digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=4561395 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=1c8db2f3-2f79-4d7b-ac30-584be6a69486 response.status="200 OK" size=4561395 url="https://harbor.registry.dns/v2/project1/blobs/sha256:373b6cb05c932efa90c2860309fbd932885aa1fe2d0b2d399d5aab81f2262c5a" DEBU[0001] do request digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=292 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" DEBU[0001] fetch response received digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=11517 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=bc41631d-96c9-4c45-8dd9-c5d5a3d3fcb7 response.status="200 OK" size=11517 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3ae285f77e5cbc3657c3c8a17f610e7ca2ac947106da3b67b5c78a359257101c" DEBU[0001] fetch response received digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=980 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=df7a8699-24d1-4207-9bdb-ba32fae5e3d9 response.status="200 OK" size=980 url="https://harbor.registry.dns/v2/project1/blobs/sha256:3f125095415cbd0c42ce8ed6fbc3335437da3d5febe45e73944af8510f6c76b4" DEBU[0001] fetch response received digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=292 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=c1ffacca-e94b-4e05-be04-6bbb16ce4237 response.status="200 OK" size=292 url="https://harbor.registry.dns/v2/project1/blobs/sha256:7764d651778cbda3c7f069f86ba147bed458dd009da9fa8adea5b10fdce6b7fd" DEBU[0001] do request digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=1001 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" DEBU[0001] do request digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=148608462 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" DEBU[0001] fetch response received digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=1001 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=41d4d7d9-3af5-40f8-ac52-a983f54149df response.status="200 OK" size=1001 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd6aea968a24fdb0a684cb364028d04b7bd49b696d44142e96dd580d7b941a10" DEBU[0001] fetch response received digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=148608462 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=6ec16589-6746-4e8e-af86-bf2fca7aea7e response.status="200 OK" size=148608462 url="https://harbor.registry.dns/v2/project1/blobs/sha256:fd3f1f3d715a631eef5b404d02efd10954d94660c657ac72fda312281b4f42bb" DEBU[0001] do request digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.6.19 request.method=GET size=5118 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" DEBU[0001] fetch response received digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=5118 response.header.content-type=application/octet-stream response.header.date="Tue, 25 Jul 2023 12:34:48 GMT" response.header.docker-content-digest="sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=6c3bc69e-38e7-4d58-8de0-ec1a3072586b response.status="200 OK" size=5118 url="https://harbor.registry.dns/v2/project1/blobs/sha256:d58d8d59a33b705299f400281243e7365bd024273d5c01769ce71feef9c10e74" DEBU[0001] do request digest="sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=2097152-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=156125588 url="https://harbor.registry.dns/v2/project1/blobs/sha256:2359ebe1d7bc809faa718538e4ec1a4bfec0ad35d591a88a8031a3dabeb5e9a1" ctr: failed to read expected number of bytes: unexpected EOF

[fuweid]

layer-sha256:f868ac632a6f7994c589ce631afb99014698a783e173144bf4740e318303e992: downloading |++++++++++++++++++++++++++++++++------| 214.0 Mi/250.5 MiB

@charugarg2810 I think there is some wrong with this blob. Would you mind to use ctr -n k8s.io leases delete leaky lease first and retry it? thanks

[charugarg2810]

The thing is it's an intermittent issue and is happening for multiple images. On some retries (~ 10-15 times) it gets auto-resolved.

[charugarg2810]

Below is the debug output for some different image: ctr --debug -n k8s.io image pull $image --user user1 Password: DEBU[0001] fetching image="harbor.registry.dns/project2:3684-cfafb9c28" DEBU[0001] resolving host=harbor.registry.dns DEBU[0001] do request host=harbor.registry.dns request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=HEAD url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] fetch response received host=harbor.registry.dns response.header.connection=keep-alive response.header.content-length=192 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jul 2023 07:12:13 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.16.1 response.header.set-cookie="sid=aa9dcd5312a404b7e8bfc8dc4f155a2e; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-request-id=a795fab4-841c-4567-a84f-206c7f2daaca response.status="401 Unauthorized" url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] Unauthorized header="Basic realm=\"harbor\"" host=harbor.registry.dns DEBU[0001] do request host=harbor.registry.dns request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.19 request.method=HEAD url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] fetch response received host=harbor.registry.dns response.header.connection=keep-alive response.header.content-length=743 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 20 Jul 2023 07:12:14 GMT" response.header.docker-content-digest="sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994\"" response.header.server=nginx/1.16.1 response.header.set-cookie="sid=e217064197c1ad5ab47e3b4a2adca8bb; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=b835fd09-a879-4248-9fb5-492fbbaad07a response.status="200 OK" url="https://harbor.registry.dns/project1/manifests/3684-cfafb9c28" DEBU[0001] resolved desc.digest="sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994" host=harbor.registry.dns DEBU[0001] fetch digest="sha256:df0f0bca49e367088b4985fcc32d5cb0abf5a15db1c531d6a82c4b06d9eb4994" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=743 DEBU[0001] fetch digest="sha256:5dc741397828b0a3c5d40b843a39a6fe78a3fc08c52f98fdc75d829161338fa7" mediatype=application/vnd.docker.distribution.manifest.v2+json size=5562 DEBU[0001] fetch digest="sha256:5d0f5cb39866238d754e5aa8277ee9654c7663c18928d053021e5b67235c3f4f" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=159 DEBU[0001] fetch digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=204219065 DEBU[0001] fetch digest="sha256:0862d060a21322221192c0217ea66bf9f43f6b9ebbba874cc6fa5aeb3d64bc31" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=5460683 DEBU[0001] fetch digest="sha256:0c2892dfbf98327e72cce129ac1201e5df5b14b97bb36e753e5a0b121d31b293" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=13686681 DEBU[0001] fetch digest="sha256:e66dd0b135f39400f503752b1f1f8b27743eac7db2efa92cf76ff54acdb0083c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1274731 DEBU[0001] fetch digest="sha256:03688cf8216857c77f07905f4a93f1dd5145f32a3d9bc1084719aa4b6e7f6dce" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4268383 DEBU[0001] fetch digest="sha256:9d361db69dbdf93606f541b86c7d04dd72c4f390b6c73712b21732daa2016095" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2419117 DEBU[0001] fetch digest="sha256:e8aae28818c31cd4cde3387c229fbab0255b21bdf14fabe61681eda6510be1ce" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=921644 DEBU[0001] fetch digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=193390970 DEBU[0001] fetch digest="sha256:4ddda290423598192b382b88f44cc18d2c76c05f20ffff871424024f58e1791c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=260 DEBU[0001] fetch digest="sha256:24ead560a6c43a1632281366a49c39bcd11a25ce37575e8dc98e885ccff53cce" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=263511 DEBU[0001] fetch digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=148608514 DEBU[0001] fetch digest="sha256:64a523d3310c30b60c073e29ede5429ce136cd2448cf50a2cec5f1f931b5a1e9" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1085 DEBU[0001] fetch digest="sha256:964cd7fa8adf4b7670619524a55ff185687c274cdb3ff778059bb82ee994e9a3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6551 DEBU[0001] fetch digest="sha256:868cd957f1d366a1193d16fcf3f3920fa93fd020e5b19ea82b8099214456c6f6" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6922 DEBU[0001] fetch digest="sha256:e5c8d896e5d455eab4b94c49a82d6a3837b44363e702b4128ed0a5f88e260dcf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=290 DEBU[0001] fetch digest="sha256:d9c9f3854d13b863781575605521d72fff338e2cce6b73986448a4313af1e625" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4818075 DEBU[0001] fetch digest="sha256:070da07b1bf535bdfc2e7f9b70669d58cb61e1744c437917ea911bc007767105" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1348 DEBU[0001] fetch digest="sha256:fa539d310f66603be77222b7950e77f164d7ddc102b6d64434483c997144af18" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=12326034 DEBU[0001] fetch digest="sha256:c58ba5199dfd2e9135e3f9381c56b8f7068475b982155c42fbdd31b6b72e8873" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6928 DEBU[0001] fetch digest="sha256:7b3aba51fd01a060fdd122a2317d78de0463d9ffbf97c267051b09411d6e859c" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=40917426 DEBU[0001] fetch digest="sha256:9ee2c1ac3140e49bea6912b70b2bdfffe08e0450d89d885115abff5f5b9ec3d6" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=4561393 DEBU[0001] fetch digest="sha256:06556d2c8f3fc9205db5f6c8ffaddb6811cb8c378ef452b03c2ca0c97673b858" mediatype=application/vnd.docker.container.image.v1+json size=9948 DEBU[0001] fetch digest="sha256:56f3752ba3fc19b9baf5228ee50dc8b23853f5f02ed41135ac08dadc095da587" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=6711223 DEBU[0001] fetch digest="sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=32 DEBU[0001] fetch digest="sha256:782a561fc530865ea2df30e7fcf23ad30e318c3038609bafbbf6b3fd53ec4e8d" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=93 DEBU[0001] do request digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=204219065 url="https://harbor.registry.dns/project1/blobs/sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" DEBU[0001] do request digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=193390970 url="https://harbor.registry.dns/project1/blobs/sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" DEBU[0002] do request digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=31457280-" request.header.user-agent=containerd/1.6.19 request.method=GET size=148608514 url="https://harbor.registry.dns/project1/blobs/sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" DEBU[0002] do request digest="sha256:fa539d310f66603be77222b7950e77f164d7ddc102b6d64434483c997144af18" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=12326034 url="https://harbor.registry.dns/project1/blobs/sha256:fa539d310f66603be77222b7950e77f164d7ddc102b6d64434483c997144af18" DEBU[0002] fetch response received digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31536000" response.header.connection=keep-alive response.header.content-length=117151234 response.header.content-range="bytes 31457280-148608513/148608514" response.header.content-type=application/octet-stream response.header.date="Thu, 20 Jul 2023 07:12:14 GMT" response.header.docker-content-digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" response.header.docker-distribution-api-version=registry/2.0 response.header.etag="\"sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea\"" response.header.server=nginx/1.16.1 response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.x-request-id=47250330-6e4d-4e32-897b-22e76bfe435e response.status="206 Partial Content" size=148608514 url="https://harbor.registry.dns/project1/blobs/sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" DEBU[0002] do request digest="sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=33554432-" request.header.user-agent=containerd/1.6.19 request.method=GET size=148608514 url="https://harbor.registry.dns/project1/blobs/sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea" DEBU[0003] fetch response received digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=192 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jul 2023 07:12:15 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.16.1 response.header.set-cookie="sid=11c5295a976eca3a9624aedfcd02c8b5; Path=/; HttpOnly; Secure" response.header.set-cookie.1="sid=e989ad272a2829e6141415413b8bec71; Path=/; HttpOnly; Secure" response.header.set-cookie.2="sid=3adfc5eb029ab1f05c8eaad6b69a82a3; Path=/; HttpOnly;Secure" response.header.set-cookie.3="sid=d9e8834c3ebc4e9359541f568ef6cc27; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-request-id=c077a6c7-9dbb-4d1c-9ba2-debfb9abe41b response.status="401 Unauthorized" size=204219065 url="https://harbor.registry.dns/project1/blobs/sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" DEBU[0003] Unauthorized digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" header="Basic realm=\"harbor\"" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=204219065 DEBU[0003] do request digest="sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=204219065 url="https://harbor.registry.dns/project1/blobs/sha256:4dab31e976054ef5783dff15a73cc907246312e0d662ea8287363c46b6ab19fb" DEBU[0003] fetch response received digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.connection=keep-alive response.header.content-length=192 response.header.content-type="application/json; charset=utf-8" response.header.date="Thu, 20 Jul 2023 07:12:15 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.16.1 response.header.set-cookie="sid=a35e264d6dbe8d99f072770c3e538b16; Path=/; HttpOnly; Secure" response.header.set-cookie.1="sid=c88d40c0bda300f38c30cda15e69627e; Path=/; HttpOnly; Secure" response.header.set-cookie.2="sid=a318f94af285e9a880c461a27b8ef271; Path=/; HttpOnly;Secure" response.header.set-cookie.3="sid=81cd99e2cd6a6b39f4bafae5dc36b283; Path=/; HttpOnly; Secure" response.header.strict-transport-security="max-age=31536000; includeSubDomains" response.header.www-authenticate="Basic realm=\"harbor\"" response.header.x-request-id=38def421-0688-48ca-977b-364be7fae2ca response.status="401 Unauthorized" size=193390970 url="https://harbor.registry.dns/project1/blobs/sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" DEBU[0003] Unauthorized digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" header="Basic realm=\"harbor\"" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=193390970 DEBU[0003] do request digest="sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.range="bytes=6291456-" request.header.user-agent=containerd/1.6.19 request.method=GETsize=193390970 url="https://harbor.registry.dns/project1/blobs/sha256:2ca5887305e5d5fe7954127278a0d9640c2b048c78b5b63b7465ce2dade3a412" ctr: failed to read expected number of bytes: unexpected EOF

[fuweid]

On some retries (~ 10-15 times) it gets auto-resolved.

So, I think it maybe related to your registry. :(

[charugarg2810]

The only concern is it's working fine with Docker. Only Conatinerd is failing.

[djdongjin]

Maybe try containerd v1.6.23 https://github.com/containerd/containerd/tree/v1.6.23 and see if the issue still exists.

Context: we got EOF errors before (not sure if it's the same root cause), and made a fix https://github.com/containerd/containerd/pull/8800 which should be included in 1.6.23.

[charugarg2810]

I tried but with no success.

root@ip-172-21-33-56 ~]# /usr/local/bin/ctr -n k8s.io image pull $image --user usr1
Password:
harbor.registry.dns/project1:3699-a809bb499:                resolved       |++++++++++++++++++++++++++++++++++++++|
index-sha256:d6a6ad06523b574f25306250670c90ac5316ecb6d0978867a783b7750b1f7ff1:    done           |++++++++++++++++++++++++++++++++++++++|
manifest-sha256:922f3e2012b77f830697a58603fc2ec6e9f3ea5e4605dba61b480ebdf5bd6601: done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:03688cf8216857c77f07905f4a93f1dd5145f32a3d9bc1084719aa4b6e7f6dce:    done           |++++++++++++++++++++++++++++++++++++++|
config-sha256:444fa77c7578f20995274254c12a615ab06041ffbb155242d19944978185ea59:   done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:0f6a07685401b9afe3b0791681ad041b69f8021c941c174b4520a629101bf7b3:    downloading    |+-------------------------------------|  6.0 MiB/194.8 MiB
layer-sha256:7b3aba51fd01a060fdd122a2317d78de0463d9ffbf97c267051b09411d6e859c:    downloading    |+++++++++++++++++++++-----------------| 22.0 MiB/39.0 MiB
layer-sha256:0c2892dfbf98327e72cce129ac1201e5df5b14b97bb36e753e5a0b121d31b293:    downloading    |+++++++++++++++++---------------------|  6.0 MiB/13.1 MiB
layer-sha256:eeda2982daa94b080aa882e79942ca697594994a32302ab316e8f8fd80048592:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9c7ded1cc7e593c8a488f25d5272993352c1f9724a184cf4730ecf332d9ab635:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:e8aae28818c31cd4cde3387c229fbab0255b21bdf14fabe61681eda6510be1ce:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3315ac82074d0ef4e31a0c74a82d68e07c28f41d750836fc4806496b0cae1d14:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:13c22b252c64e1120ce74d4e091e29940e8480532f4d55971f1cab5757b8e862:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4ddda290423598192b382b88f44cc18d2c76c05f20ffff871424024f58e1791c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:3e7ec20fc0634a378976a8edc0e82354446819925dc0e5a287c19e2601aa823e:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:12d7196581569a9b12b1fd774cc20c3dc07154c1d3145d324749144db3ba053c:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9bff18081715534d9019a27f417f0fe5e2b50cdcd25cbdf5fff8eb5c9bd0a3ea:    downloading    |--------------------------------------|  2.0 MiB/141.7 MiB
layer-sha256:e0d2f29fcf852715b0534d30ae5ef14b4dd774947b479bf36486f08e6635e492:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1:    waiting        |--------------------------------------|
layer-sha256:c8a5cd881d8ae893c2ed54fc96bc2f0703c9271056488acce23ad317d930bd48:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:964cd7fa8adf4b7670619524a55ff185687c274cdb3ff778059bb82ee994e9a3:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:b86064b015f08be41226d7a586f039711a357b498a5d040f83d144becf902020:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:44d2315972efb3f4d353b27393441271c3a3f7b56531fb3e4af5c3274e256732:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bbbdedd4d4581a125f50348660142fed0990444377c0136392023a26b6c243ee:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:bba9d6338fff27ce8138e8085d4c09e5816f72fc1b2c2f0163800a33e9eca774:    downloading    |+-------------------------------------|  6.0 MiB/184.4 MiB
layer-sha256:e5c8d896e5d455eab4b94c49a82d6a3837b44363e702b4128ed0a5f88e260dcf:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:618e0d39adef9038f7437a861ca250f529291fd03af0c3e58597f1a844f06ae9:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:9ee2c1ac3140e49bea6912b70b2bdfffe08e0450d89d885115abff5f5b9ec3d6:    done           |++++++++++++++++++++++++++++++++++++++|
layer-sha256:dff02559314f94cc184ecd2858a9a7435faf549a8b2c50789e371b90b3b1b63e:    done           |++++++++++++++++++++++++++++++++++++++|
elapsed: 0.6 s                                                                    total:  83.1 M (137.9 MiB/s)
ctr: failed to read expected number of bytes: unexpected EOF
[root@ip-172-21-33-56 ~]#
[root@ip-172-21-33-56 ~]#
[root@ip-172-21-33-56 ~]# ctr --version
ctr github.com/containerd/containerd v1.6.23
[root@ip-172-21-33-56 ~]#

[d4n13lbc]

Hi. I have the same issue with EKS 1.25 and containerd version 1.6.19

[charugarg2810]

@d4n13lbc you too using the Harbor Docker registry?

[starpipi]

@charugarg2810 I use Harbor Docker registry too. I can pull images with ctr or nerdctl,But crictl can't pull. Just shown EOF I tried containerd version 1.7.5 and 1.6.23. It's the same.

[srinathrangaramanujam]

We have the same issue eith eks 1.25.

[kmursk]

Fatal error... New versions of kubernetes use crictl only, but crictl is not able to pull images by https. While this issue is not resolved, the single solution i had found is to switch harbor to work through http and force containerd do the same.

[rbjorklin]

We are seeing the same intermittent issue with JFrog Artifactory.

[rbjorklin]

Might this be a regression in containerd? In migrating our container runtime we made a rather significant version bump. We used to be on Docker 20.10.11 which bundled containerd=1.4.12 and jumped all the way to containerd=1.6.20.

What's the situation for everyone else who have chimed in on this issue?

EDIT: For what it's worth it looks like there is an integration test checking for this error.

[xiang90]

Hit the same issue with containerd 1.7.11 with AWS ECR

See

  Warning  Failed            58m                 kubelet            Failed to pull image "xxx": rpc error: code = Unavailable desc = error reading from server: EOF

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 7 days unless new comments are made or the stale label is removed.

[rbjorklin]

Not stale.

[yulinor]

The same error is reported, distribution v2.8.3 is used as a cache pull warehouse, and the node uses containerd 1.6.23. Occasionally there will be an image that cannot be pulled on all nodes, but it can be pulled after restarting the distribution.

I strongly suspect that there is a problem with distribution, but the info/debug logs of the service are all Get HTTP requests, filesystem.getcontent,filesystem.Reader, filesystem.putcontent, response completed, filesystem.Delete repeatedly,no error log output

[artemijan]

I noticed the same issue in our build system, the root cause was that we messed up with docker cache and loaded (docker load) wrong images to build main app.