search

estesp / manifest-tool

Command line tool to create and query container image manifest list/indexes
Apache License 2.0
747 stars 94 forks source link

Is 2.0.0 broken for Quay.io? #156

Closed tboerger closed 2 years ago

tboerger commented 2 years ago

I'm using the manifest-tool already quite long and never had any issues with it until I updated to 2.0.0. With this version I'm receiving an error while trying to push the manifest to Quay:

time="2022-03-07T11:34:32Z" level=fatal msg="Error pushing manifest list/index to registry: sha256:8d63af9ef7ceccfc644d2507fae12251cfbeb05070e1af80525cc6cf6721311a: failed commit on ref \"manifest-sha256:a31e11ed2de7e55e5c9eb5697b88913a4d8f1209283694e04b8399baaa9a4009\": cannot reuse body, request must be retried"

Is there anything I can fix on my side? Or is Quay really currently broken with manifest-tool?

estesp commented 2 years ago

I've tried to reproduce this without success; can you provide some more data; maybe with --debug? My local attempts with both main's HEAD commit (a few vendored updates since 2.0 release) as well as the exact v2.0.0 binary are below. I'm only including the push of the final manifest list entry with --debug to validate it pushed to quay:

ubuntu@devvm:~/go/src/github.com/estesp/manifest-tool/integration$ ./test-registry.sh quay.io/estesp
Warning: some commands will fail if you are not authenticated to quay.io/estesp
[ skipping lots of debug output ]
DEBU[0002] checking and pushing to                       digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0002] do request                                    digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.accept="application/vnd.docker.distribution.manifest.list.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0003] fetch response received                       digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=132 response.header.content-type=application/json response.header.date="Mon, 07 Mar 2022 18:48:04 GMT" response.header.server=nginx/1.14.1 response.status="404 Not Found" size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0003] do request                                    digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json request.header.user-agent=containerd/1.5.9+unknown request.method=PUT size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
DEBU[0003] fetch response received                       digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" mediatype=application/vnd.docker.distribution.manifest.list.v2+json response.header.content-length=2 response.header.content-type="text/html; charset=utf-8" response.header.date="Mon, 07 Mar 2022 18:48:04 GMT" response.header.docker-content-digest="sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" response.header.location="https://quay.io/v2/estesp/alpine/manifests/sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="201 Created" size=1272 url="https://quay.io/v2/estesp/alpine/manifests/latest"
Digest: sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7 1272

I then inspected this new image (just FYI: I deleted all prior tags/shas from the "estesp" repo before the above step, and also tested with existing tags in place) to verify it pushed properly:

$ manifest-tool inspect quay.io/estesp/alpine:latest
Name:   quay.io/estesp/alpine:latest (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:ab6a6c4d9ae12b0040b5d19c115f1666590c18e90124aa69ed95a802604b37d7
 * Contains 4 manifest references:
[1]     Type: application/vnd.docker.distribution.manifest.v2+json
[1]   Digest: sha256:2042a492bcdd847a01cd7f119cd48caa180da696ed2aedd085001a78664407d6
[1]   Length: 528
[1] Platform:
[1]    -      OS: linux
[1]    -    Arch: ppc64le
[1] # Layers: 1
     layer 01: digest = sha256:159b5dcb1717c815c76ff5ea1db730e18e8609c9090238e43282856db9e71f47

[2]     Type: application/vnd.docker.distribution.manifest.v2+json
[2]   Digest: sha256:e7d88de73db3d3fd9b2d63aa7f447a10fd0220b7cbf39803c803f2af9ba256b3
[2]   Length: 528
[2] Platform:
[2]    -      OS: linux
[2]    -    Arch: amd64
[2] # Layers: 1
     layer 01: digest = sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3

[3]     Type: application/vnd.docker.distribution.manifest.v2+json
[3]   Digest: sha256:49e322ab6690e73a4909f787bcbdb873631264ff4a108cddfd9f9c249ba1d58e
[3]   Length: 528
[3] Platform:
[3]    -      OS: linux
[3]    -    Arch: s390x
[3] # Layers: 1
     layer 01: digest = sha256:d6baca485f3d0f7c77221be60fbef5db014a5ef9d8f53db4a310c947c690d189

[4]     Type: application/vnd.docker.distribution.manifest.v2+json
[4]   Digest: sha256:6ace9a4351a3f02894c273cee7adc519318d6186f0d364fa37e418394576c5ef
[4]   Length: 528
[4] Platform:
[4]    -      OS: linux
[4]    -    Arch: arm64
[4]    - Variant: v8
[4] # Layers: 1
     layer 01: digest = sha256:bc1d962af60541fd028aaa6f21d62a662d718b3571577acf3fb65c04a61b1150
mthalman commented 2 years ago

I have the exact opposite problem with Quay. It works with 2.0.0 but fails with 1.0.3:

$ manifest-tool inspect quay.io/centos/centos:stream9-development
time="2022-03-07T19:30:15Z" level=error msg="Error trying v2 registry: unsupported manifest format"
time="2022-03-07T19:30:15Z" level=fatal msg="unsupported manifest format"
estesp commented 2 years ago

I have the exact opposite problem with Quay. It works with 2.0.0 but fails with 1.0.3:

Yeah, I didn't have OCIv1 image spec support in the 1.x branch at all :( The CentOS images are true OCI images:

tboerger commented 2 years ago

I have executed the manifest-tool now manually with an older version of my images. I tried it with the existing manifest and I also tried it after deleting the manifest, same results:

$ docker run --rm mplatform/manifest-tool --debug --username $QUAY_USERNAME --password $QUAY_PASSWORD push from-args --platforms linux/amd64,linux/arm64,linux/arm/v6 --template quay.io/webhippie/alpine:3.6-ARCH --target quay.io/webhippie/alpine:3.6
time="2022-03-09T08:00:18Z" level=info msg="Retrieving digests of member images"
time="2022-03-09T08:00:18Z" level=debug msg=resolving host=quay.io
time="2022-03-09T08:00:18Z" level=debug msg="do request" host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD url="https://quay.io/v2/webhippie/alpine/manifests/3.6-amd64"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" host=quay.io response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/webhippie/alpine/manifests/3.6-amd64"
time="2022-03-09T08:00:19Z" level=debug msg=resolved desc.digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:3fbd95657a45dfed369740b70575f19dd5363f96474dbc2f57ed4367c637bcf2"
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=632 response.header.content-length=2925 response.header.content-type=binary/octet-stream response.header.date="Wed, 09 Mar 2022 07:49:48 GMT" response.header.etag="\"506d261ba5c9f1806e97cc85ae78e377-1\"" response.header.last-modified="Wed, 09 Mar 2022 03:35:41 GMT" response.header.server=AmazonS3 response.header.via="1.1 1a45d1e1304c39dfa9b034c2308f4976.cloudfront.net (CloudFront)" response.header.x-amz-cf-id="YjMRJIx7uHMhW72Hubr1VGwZP2ZdhndcTTjYaC7bMqcHqDA8lEIGcg==" response.header.x-amz-cf-pop=MUC50-P1 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=f59qh4eKw_GAZwMOE5jAuiFP9hQD2N24 response.header.x-cache="Hit from cloudfront" response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:19Z" level=debug msg=resolving host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg="do request" host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm64"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" host=quay.io response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm64"
time="2022-03-09T08:00:19Z" level=debug msg=resolved desc.digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:19 GMT" response.header.docker-content-digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3"
time="2022-03-09T08:00:19Z" level=debug msg=fetch digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:19Z" level=debug msg="do request" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:19Z" level=debug msg="fetch response received" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=632 response.header.content-length=2925 response.header.content-type=binary/octet-stream response.header.date="Wed, 09 Mar 2022 07:49:49 GMT" response.header.etag="\"2c704921fb41bcada721b0b741ae5141-1\"" response.header.last-modified="Wed, 09 Mar 2022 03:35:56 GMT" response.header.server=AmazonS3 response.header.via="1.1 1a45d1e1304c39dfa9b034c2308f4976.cloudfront.net (CloudFront)" response.header.x-amz-cf-id="2ljActRxc-O7Q08MTeUKDnfoUf8Eu-KvTZeOqvfvZNoHwpxKM_IQNQ==" response.header.x-amz-cf-pop=MUC50-P1 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=RJGYCnMYXfQvq9cOiFcPZ.Tg3C1emQK0 response.header.x-cache="Hit from cloudfront" response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:19Z" level=debug msg=resolving host=quay.io
time="2022-03-09T08:00:19Z" level=debug msg="do request" host=quay.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" host=quay.io response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" url="https://quay.io/v2/webhippie/alpine/manifests/3.6-arm"
time="2022-03-09T08:00:20Z" level=debug msg=resolved desc.digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" host=quay.io
time="2022-03-09T08:00:20Z" level=debug msg=fetch digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=1367 response.header.content-type=application/vnd.docker.distribution.manifest.v2+json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/sha256:637e6480a460ada228d356e94fb4945068c10c71ec91dc5342b24dc79f9b0153"
time="2022-03-09T08:00:20Z" level=debug msg=fetch digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json size=2938
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=GET size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.age=631 response.header.content-length=2938 response.header.content-type=binary/octet-stream response.header.date="Wed, 09 Mar 2022 07:49:50 GMT" response.header.etag="\"a31a9b409a4ebf882ce8f47875efe380-1\"" response.header.last-modified="Wed, 09 Mar 2022 03:36:11 GMT" response.header.server=AmazonS3 response.header.via="1.1 1a45d1e1304c39dfa9b034c2308f4976.cloudfront.net (CloudFront)" response.header.x-amz-cf-id="282hsOOJzaPoowT1jITQuGnUSXZ-EhLsqjr6sgSvtaWCrpVjGBLQZQ==" response.header.x-amz-cf-pop=MUC50-P1 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=1pyb_YPsRMxiuWPD2Fq0_KwZP3JZboWv response.header.x-cache="Hit from cloudfront" response.status="200 OK" size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11763211
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11763211 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=11763211 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2017774
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=2017774 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2017774 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1986249
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1986249 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1986249 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1436 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11651310
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11651310 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=954
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=11651310 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=954 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1936225
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1936225 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11870737
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json size=2925
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1936225 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10964820
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10179823
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10964820 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=10964820 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=1436
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10301040
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json size=2938
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10179823 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=10179823 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=954 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=10301040 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=10301040 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=11870737 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=11870737 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3"
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json request.header.accept="application/vnd.docker.container.image.v1+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip request.header.accept="application/vnd.docker.image.rootfs.diff.tar.gzip, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2925 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:9e8c361252e68d2dd89150eff2242a8f67960d4f42b3c87bc75691aa0e03878d"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=11763211 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=11763211 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e6c5a393d8d1e3faa732bc62622ad2e8dc8206d96f297892828f45dc248a7d96"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=1436 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1436 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:a831600a8b89a2c7966e74ba39dbe8445a3f48485b515a98eb4064a65a14e74b"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=11651310 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=11651310 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:98951c2dfabd790a6bda492ef5782234eff0bedaa02b366d562549436b592800"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2017774 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2017774 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:5a3ea8efae5d0abb93d2a04be0a4870087042b8ecab8001f613cdc2a9440616a"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2925 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2925 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:022c77f5ae044caad4786fa51b28cdb27e5406120664c62f05b41c336138ee38"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=1986249 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1986249 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:cc0316aec136ecacf8d375c9bff734812dde163b2edcab97c1c87ef50809dfbf"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" mediatype=application/vnd.docker.container.image.v1+json response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=2938 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=2938 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:ea18fe12e67f3ad677e83d3ee74bc2c72aded8e77c42811e11352fbaa9742677"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=1936225 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=1936225 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:e8f81692e76c11ec1936df19ecdb11a3c7708adcb60b481f942689ccd9474c01"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=10301040 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=10301040 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:73c6d8f222a884789be203ea7cae6386b4f54a92e9c540dd991365950cc0b2be"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=10179823 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=10179823 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:110d1638fb7c3b0f1966cf0d608617b8f8cfb795288096ddc3f70209d1556b49"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=11870737 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=11870737 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:2fa29e73267cf241395be505134404b1d337d3ba639bf1d623a33d2fcd3243f3"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=10964820 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=10964820 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:542ea154e03fb8b35b9ca62445bc090a7fe9e093c30538e0b653743b80f81cc4"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=954 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=954 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:1a286ff9b6ce51fa6bf6c94a2c68fa2d2dd1bebe5c814b9482bf3417051e26b3"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=955 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:f1e7e712b2794eb40ad5be9261c72e2b98f490ea6dbd9801136f1c36ef592fae"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.cache-control="max-age=31436000" response.header.content-length=955 response.header.content-type=application/octet-stream response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-content-digest="sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3" response.header.server=nginx/1.14.1 response.header.strict-transport-security="max-age=63072000; preload" response.header.x-frame-options=DENY response.status="200 OK" size=955 url="https://quay.io/v2/webhippie/alpine/blobs/sha256:6cce0c6f0475dff0fe222025f10d3a6f36a9102938f1ec483b63e5f28b4fa5e3"
time="2022-03-09T08:00:20Z" level=debug msg=push digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:20Z" level=debug msg="checking and pushing to" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.accept="application/vnd.docker.distribution.manifest.v2+json, */*" request.header.user-agent=containerd/1.5.9+unknown request.method=HEAD size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=129 response.header.content-type=application/json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.server=nginx/1.14.1 response.status="404 Not Found" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="do request" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json request.header.content-type=application/vnd.docker.distribution.manifest.v2+json request.header.user-agent=containerd/1.5.9+unknown request.method=PUT size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg="fetch response received" digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" mediatype=application/vnd.docker.distribution.manifest.v2+json response.header.content-length=112 response.header.content-type=application/json response.header.date="Wed, 09 Mar 2022 08:00:20 GMT" response.header.docker-distribution-api-version=registry/2.0 response.header.server=nginx/1.14.1 response.header.www-authenticate="Bearer realm=\"https://quay.io/v2/auth\",service=\"quay.io\",scope=\"repository:webhippie/alpine:pull,push\"" response.status="401 Unauthorized" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"
time="2022-03-09T08:00:20Z" level=debug msg=Unauthorized digest="sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3" header="Bearer realm=\"https://quay.io/v2/auth\",service=\"quay.io\",scope=\"repository:webhippie/alpine:pull,push\"" mediatype=application/vnd.docker.distribution.manifest.v2+json size=1367
time="2022-03-09T08:00:20Z" level=fatal msg="Error pushing manifest list/index to registry: sha256:0c2d7d615df8b704e5aaf8fe62e90bf33ec7a0bd0fefca0bc54b77b5d9d720d5: failed commit on ref \"manifest-sha256:316470549e4eb403bf2c67c454cdc644b99dba23d625c299e75ebd00a02fe5e3\": cannot reuse body, request must be retried"

For now I will do a downgrade to latest 1.x.x release to get my pipelines working again, but I would love to use the new version when it works, of course. The same image works fine with 1.0.3:

$ docker run --rm mplatform/manifest-tool:v1.0.3 --username $QUAY_USERNAME --password $QUAY_PASSWORD push from-args --platforms linux/amd64,linux/arm64,linux/arm/v6 --template quay.io/webhippie/alpine:3.6-ARCH --target quay.io/webhippie/alpine:3.6
Digest: sha256:112d1f640d44a844a65b4465183d409b6ac2cdb6ca8038698e93976855712d3c 1079
estesp commented 2 years ago

Looks like #122 is finding the same issue; I don't get the 404 that I see near the end of your debug output when it's trying to assemble the final manifest in the registry:

response.status="404 Not Found" size=1367 url="https://quay.io/v2/webhippie/alpine/manifests/3.6"

and it's from there that the reply/response cycle may actually be reusing a body incorrectly when it triggers an auth flow?

tboerger commented 2 years ago

If you need any further debug information just let me know :)

estesp commented 2 years ago

I was able to reproduce it; there are 2 issues at play--one is the known "cannot reuse body" issue which we have someone assigned to work on and fix in the containerd registry push handler. When that fix is available it should solve the overall problem of retry with auth during a push.

The second is that it looks like maybe there is something with token scopes/auth that makes this only reproducible when you use the same repository for the source(s)/target of the manifest list/index. For example, I had images like quay.io/estesp/amd64_alpine:3.11 that were then populating a manifest list in quay.io/estesp/alpine:3.11 (meaning 5 different repositories were being used; 4 uniquely named repos with the architecture as part of the name and the target "alpine:..." repository.

When I change my configuration to source the architectures and final list from the same repo I get the same 404 on the existence check of the manifest list, followed by a 401 to re-auth, which gets the "cannot reuse body" error.

The 404 is confusing as I would expect that to happen on the existence check HEAD request on the target manifest list in the other flow (where the source repos are all different), but it doesn't happen. Will need to dig further but at least now I can reproduce.

estesp commented 2 years ago

Worked around in #159 until containerd has a complete fix for the retry on auth challenge issue with req body reuse.

I've tested Quay and verified this fixes this issue with using manifest-tool v2 and Quay.io. Interestingly, if you set the repo to private or source any of the platform images from a private repo the 401 auth challenge comes during a "fetch" and you don't get the error. That's why I initially never saw this issue when testing.

estesp commented 2 years ago

You can test/verify with the latest release:

$ docker run --rm mplatform/manifest-tool --version
/manifest-tool version 2.0.3 (commit: 65590ecce1d4af199724d235dbb5453e10cad420)
tboerger commented 2 years ago

Thanks for the effort, I can confirm that the latest release has resolved my issues for Quay.io :)