Closed whoisac closed 1 year ago
Layers don't get compressed on disk (their contents are stored in exploded form), and the manifest's description of the layer tends to reflect that.
The problem appears to be that the layer didn't get compressed while it was being pushed, though I generally expect layer blobs to get compressed by default. Can you retry the podman push
with --log-level=debug
and show us what it says?
The logic tries try to skip copying/pushing a layer if it notices that the registry already has a copy of that layer, but in older versions it would forget to modify the MIME type for a layer in the manifest that had been compressed/decompressed/recompressed if it skipped pushing that layer. I thought we'd fixed that, though.
Thanks for explaining the layers remain in uncompressed in local storage. This podman push log:
+ podman push --log-level=debug --format v2s2 --creds KEY internal_registry/testimage:latest
INFO[0000] podman filtering at log level debug
DEBU[0000] Called push.PersistentPreRunE(podman push --log-level=debug --format v2s2 --creds KEY internal_registry/testimage:latest)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/jenkins/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/1000/containers" with "/tmp/podman-run-1000/containers" from database
DEBU[0000] Overriding tmp dir "/run/user/1000/libpod/tmp" with "/tmp/podman-run-1000/libpod/tmp" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/jenkins/.local/share/containers/storage
DEBU[0000] Using run root /tmp/podman-run-1000/containers
DEBU[0000] Using static dir /home/jenkins/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp
DEBU[0000] Using volume path /home/jenkins/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 13
INFO[0000] podman filtering at log level debug
DEBU[0000] Called push.PersistentPreRunE(podman push --log-level=debug --format v2s2 --creds KEY internal_registry/testimage:latest)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/jenkins/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Overriding run root "/run/user/1000/containers" with "/tmp/podman-run-1000/containers" from database
DEBU[0000] Overriding tmp dir "/run/user/1000/libpod/tmp" with "/tmp/podman-run-1000/libpod/tmp" from database
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/jenkins/.local/share/containers/storage
DEBU[0000] Using run root /tmp/podman-run-1000/containers
DEBU[0000] Using static dir /home/jenkins/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/podman-run-1000/libpod/tmp
DEBU[0000] Using volume path /home/jenkins/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that overlay is supported
DEBU[0000] Cached value indicated that metacopy is not being used
DEBU[0000] Cached value indicated that native-diff is usable
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Setting parallel job count to 13
DEBU[0000] Looking up image "internal_registry/testimage:latest" in local containers storage
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Trying "internal_registry/testimage:latest" ...
DEBU[0000] parsed reference into "[overlay@/home/jenkins/.local/share/containers/storage+/tmp/podman-run-1000/containers]@7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f"
DEBU[0000] Found image "internal_registry/testimage:latest" as "internal_registry/testimage:latest" in local containers storage
DEBU[0000] Found image "internal_registry/testimage:latest" as "internal_registry/testimage:latest" in local containers storage ([overlay@/home/jenkins/.local/share/containers/storage+/tmp/podman-run-1000/containers]@7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f)
DEBU[0000] Pushing image internal_registry/testimage:latest to internal_registry/testimage:latest
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux [] }
DEBU[0000] Copying source image [overlay@/home/jenkins/.local/share/containers/storage+/tmp/podman-run-1000/containers]@7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f to destination image //internal_registry/testimage:latest
DEBU[0000] Returning credentials for internal_registry/testimage from DockerAuthConfig
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000] Using "default-docker" configuration
DEBU[0000] Using file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/internal_registry
DEBU[0000] Loading registries configuration "/home/jenkins/.config/containers/registries.conf"
DEBU[0000] Using blob info cache at /home/jenkins/.local/share/containers/cache/blob-info-cache-v1.boltdb
DEBU[0000] IsRunningImageAllowed for image containers-storage:[overlay@/home/jenkins/.local/share/containers/storage]@7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Manifest has MIME type application/vnd.docker.distribution.manifest.v2+json, ordered candidate list [application/vnd.docker.distribution.manifest.v2+json]
DEBU[0000] ... will first try using the original manifest unmodified
DEBU[0000] Checking /v2/testimage/blobs/sha256:a9820c2af00a34f160836f6ef2044d88e6019ca19b3c15ec22f34afe9d73f41c
DEBU[0000] GET https://internal_registry/v2/
DEBU[0000] Checking /v2/testimage/blobs/sha256:3d5ecee9360ea8711f32d2af0cab1eae4d53140496f961ca1a634b5e2e817412
DEBU[0000] Checking /v2/testimage/blobs/sha256:d4f0e88b20938ed69d29fce5ec4382f51e799f982c8d55fa9e827be0ef6ccaf5
DEBU[0000] Checking /v2/testimage/blobs/sha256:a733283f3af5dc4e0ccaf080ab94c3b3354cea1522085478fef52bdb238ff4b1
DEBU[0000] Checking /v2/testimage/blobs/sha256:f8afc1308a31d57af185d198c3adec16a132291ebab160cd50abcbde78407c62
DEBU[0000] Checking /v2/testimage/blobs/sha256:d3ba1354d62016337c31daf46c6a1e0b6f3f24b716705acf28768b8ddf0efe11
DEBU[0000] Ping https://internal_registry/v2/ status 401
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&service=internal_registry
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&service=internal_registry
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&service=internal_registry
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&service=internal_registry
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&service=internal_registry
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&service=internal_registry
DEBU[0000] HEAD https://internal_registry/v2/testimage/blobs/sha256:d3ba1354d62016337c31daf46c6a1e0b6f3f24b716705acf28768b8ddf0efe11
DEBU[0000] HEAD https://internal_registry/v2/testimage/blobs/sha256:a9820c2af00a34f160836f6ef2044d88e6019ca19b3c15ec22f34afe9d73f41c
DEBU[0000] HEAD https://internal_registry/v2/testimage/blobs/sha256:f8afc1308a31d57af185d198c3adec16a132291ebab160cd50abcbde78407c62
DEBU[0000] HEAD https://internal_registry/v2/testimage/blobs/sha256:3d5ecee9360ea8711f32d2af0cab1eae4d53140496f961ca1a634b5e2e817412
DEBU[0000] ... not present
DEBU[0000] Trying to reuse cached location sha256:4c63d9670de300356f7ab478e97790659bf3137f055c8a0125c769c774aa7354 compressed with gzip in internal_registry/testimage
DEBU[0000] Checking /v2/testimage/blobs/sha256:4c63d9670de300356f7ab478e97790659bf3137f055c8a0125c769c774aa7354
DEBU[0000] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&scope=repository%3Atestimage%3Apull&service=internal_registry
DEBU[0002] ... not present
DEBU[0002] Trying to reuse cached location sha256:311e36338b5936256af11150cf5cd65b0f03fe091a743f883155bee788995082 compressed with gzip in internal_registry/testimage
DEBU[0002] Checking /v2/testimage/blobs/sha256:311e36338b5936256af11150cf5cd65b0f03fe091a743f883155bee788995082
DEBU[0002] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&scope=repository%3Atestimage%3Apull&service=internal_registry
DEBU[0002] ... not present
DEBU[0002] Trying to reuse cached location sha256:373bdd6964713c129bba2b5781f5dfe34a2bb42132c2c64ddbc26f6b81eb2b4e compressed with gzip in internal_registry/testimage
DEBU[0002] Checking /v2/testimage/blobs/sha256:373bdd6964713c129bba2b5781f5dfe34a2bb42132c2c64ddbc26f6b81eb2b4e
DEBU[0002] GET https://internal_registry/artifactory/api/docker/internal-docker-local/v2/token?account=andrew.chum%40xxx.com&scope=repository%3Atestimage%3Apull%2Cpush&scope=repository%3Atestimage%3Apull&service=internal_registry
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:d4f0e88b20938ed69d29fce5ec4382f51e799f982c8d55fa9e827be0ef6ccaf5
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:a733283f3af5dc4e0ccaf080ab94c3b3354cea1522085478fef52bdb238ff4b1
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:311e36338b5936256af11150cf5cd65b0f03fe091a743f883155bee788995082
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:373bdd6964713c129bba2b5781f5dfe34a2bb42132c2c64ddbc26f6b81eb2b4e
DEBU[0002] ... not present
DEBU[0002] Trying to reuse cached location sha256:7f90fadf1b0d06a2b1f4a641760ecb121dc6a0d02f9422df2302201233e38b29 compressed with gzip in internal_registry/testimage
DEBU[0002] Checking /v2/testimage/blobs/sha256:7f90fadf1b0d06a2b1f4a641760ecb121dc6a0d02f9422df2302201233e38b29
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:7f90fadf1b0d06a2b1f4a641760ecb121dc6a0d02f9422df2302201233e38b29
DEBU[0002] ... not present
DEBU[0002] Trying to reuse cached location sha256:3e6a35d6262cd6d06edc97c7259ec7d5928449685509776d212877a58deb79bb compressed with gzip in internal_registry/testimage
DEBU[0002] Checking /v2/testimage/blobs/sha256:3e6a35d6262cd6d06edc97c7259ec7d5928449685509776d212877a58deb79bb
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:3e6a35d6262cd6d06edc97c7259ec7d5928449685509776d212877a58deb79bb
DEBU[0002] ... not present
DEBU[0002] Trying to reuse cached location sha256:d43d4ddd072e4276fdb28a609b56c77603132775e8488170451595c6c394dbc8 compressed with gzip in internal_registry/testimage
DEBU[0002] Checking /v2/testimage/blobs/sha256:d43d4ddd072e4276fdb28a609b56c77603132775e8488170451595c6c394dbc8
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:d43d4ddd072e4276fdb28a609b56c77603132775e8488170451595c6c394dbc8
DEBU[0002] ... already exists
DEBU[0002] Skipping blob sha256:d4f0e88b20938ed69d29fce5ec4382f51e799f982c8d55fa9e827be0ef6ccaf5 (already present):
Copying blob 7f90fadf1b0d skipped: already exists
DEBU[0002] Checking /v2/testimage/blobs/sha256:0330cf831ddc04713ef8803fbc5d8bfadc4064a100297a2b7fb1cab53e573b7f
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:0330cf831ddc04713ef8803fbc5d8bfadc4064a100297a2b7fb1cab53e573b7f
DEBU[0002] ... already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
DEBU[0002] HEAD https://internal_registry/v2/testimage/blobs/sha256:4c63d9670de300356f7ab478e97790659bf3137f055c8a0125c769c774aa7354
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
DEBU[0002] Checking /v2/testimage/blobs/sha256:76394860210a7d100adbbef6790fe8ceade4181925eb8f87e5582608e96d2252
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 [--------------------------------------] 0.0b / 461.1MiB
Copying blob 311e36338b59 skipped: already exists
DEBU[0004] ... not present
DEBU[0004] exporting filesystem layer "5cb1805a50a7bd1ebd389e559799e83d3ed688da70042d96b184b4d3013d0858" without compression for blob "sha256:8de52563936963f33744cbb33003387cea688063c08cc6a2ecad65899587e182"
DEBU[0004] No compression detected
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 [--------------------------------------] 0.0b / 461.1MiB
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 [--------------------------------------] 0.0b / 461.1MiB
Copying blob 311e36338b59 skipped: already exists
Copying blob 8de525639369 done
Copying blob 07b17deed702 [--------------------------------------] 8.0b / 12.7MiB
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 [--------------------------------------] 0.0b / 461.1MiB
Copying blob 311e36338b59 skipped: already exists
Copying blob 8de525639369 done
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 [--------------------------------------] 8.0b / 461.1MiB
Copying blob 311e36338b59 skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 [--------------------------------------] 8.0b / 461.1MiB
Copying blob 311e36338b59 skipped: already exists
Copying blob 8de525639369 done
Copying blob 07b17deed702 done
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 7f90fadf1b0d skipped: already exists
Copying blob 373bdd696471 skipped: already exists
Copying blob 3e6a35d6262c skipped: already exists
Copying blob d43d4ddd072e skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
Copying blob d738238a10b6 done
Copying blob 311e36338b59 skipped: already exists
Copying blob 8de525639369 done
Copying blob 07b17deed702 done
Copying blob 4c63d9670de3 skipped: already exists
Copying blob 82cffe18899f done
Copying blob 6586d341a7e1 skipped: already exists
Copying blob 6c4c71e4ead6 done
Copying blob a6dc5b9b681c done
Copying blob 2184733c418e skipped: already exists
DEBU[0012] exporting opaque data as blob "sha256:7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f"
DEBU[0012] No compression detected
DEBU[0012] Using original blob without modification
DEBU[0012] Checking /v2/testimage/blobs/sha256:7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f
DEBU[0012] HEAD https://internal_registry/v2/testimage/blobs/sha256:7ede2eca7fe25085cac0d8ccab726ed83a572e80a92abf3092545b1b403c4e6f
Copying config 7ede2eca7f [--------------------------------------] 8.0b / 10.8KiB
DEBU[0013] ... not present
DEBU[0013] Uploading /v2/testimage/blobs/uploads/
Copying config 7ede2eca7f [--------------------------------------] 8.0b / 10.8KiB
Copying config 7ede2eca7f done
DEBU[0013] PUT https://internal_registry/v2/internal-docker-local/testimage/blobs/uploads/4395e7f8-fac2-41b1-945b-33776a741111?digest=sha256%3A7ede2eca7fe25085cac0d8ccab726ed83
Copying config 7ede2eca7f done
Copying config 7ede2eca7f done
Writing manifest to image destination
DEBU[0013] PUT https://internal_registry/v2/testimage/manifests/latest
Storing signatures
DEBU[0014] Called push.PersistentPostRunE(podman push --log-level=debug --format v2s2 --creds KEY internal_registry/testimage:latest)
This is the inspection:
+ echo '*** skopeo inspect --raw --creds KEY docker://internal_registry/testimage:latest | jq | grep mediaType'
*** skopeo inspect --raw --creds KEY docker://internal_registry/testimage:latest | jq | grep mediaType
+ skopeo inspect --raw --creds KEY docker://internal_registry/testimage:latest
+ jq
+ grep mediaType
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"mediaType": "application/vnd.docker.container.image.v1+json",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
+ echo '*** skopeo inspect --creds KEY docker://internal_registry/testimage:latest | jq .Digest'
*** skopeo inspect --creds KEY docker://internal_registry/testimage:latest | jq .Digest
+ jq .Digest
+ skopeo inspect --creds KEY docker://internal_registry/testimage:latest
"sha256:813f9435164cdd6b4c9f1cb12df6d1ff2f59f9e6ad37d7e30d2cbb5cfcd7bc43"
From what you said, the internal registry could already have the uncompressed layer which caused the podman push not to compress. However, we have no control on the destination registry. May be an old version of podman pushed there. Can the code be changed so that it always compress layers during pushing?
A friendly reminder that this issue had no activity for 30 days.
Anyone has any idea?
@nalind @vrothberg @mtrmac Could you answer @whoisac question.
The inspect output is filtered out not to contain digests, which makes it difficult to tell for sure what is going on here.
In principle, if an uncompressed version exists on the registry, the push is going to use it instead of compressing. The copy code tries fairly hard to avoid copies, and it does not treat uncompressed data specially in that regard.
Currently there is no reasonable way to override that behavior. (An unreasonable way might be to push with encryption, and then decrypt the image, possibly to a different repo, but I didn’t test that.)
As a workaround right now, I think running a local registry with no previous state or images in a container, pushing to that registry, and skopeo copy
to the final destination, would work.
@mtrmac , is it possible remove that "uncompressed version" in the registry? How would the uncompressed layer exist in the first place in ? As I understand it now, "podman push" will compress first and push if the destination registry doesn't have that layer cached. Is it correct?
is it possible remove that "uncompressed version" in the registry?
It might be possible ( https://github.com/distribution/distribution/blob/main/docs/spec/api.md#deleting-a-layer ) but I don’t know of any software that directly exposes this.
Maybe delete all images that refer to that blob, and run a registry’s garbage collection process, if any.
How would the uncompressed layer exist in the first place
An explicit request (e.g. skopeo copy --preserve-digests
) or maybe an old bug. I agree it should not usually happen.
A friendly reminder that this issue had no activity for 30 days.
Since this does not seem to be a buildah issue, I am closing. Feel free to continue the conversation here.
Description
All layers are uncompressed, except one. After podman push to an internal artifactory registry.
All layers are compressed, except one. If pushing to quay.io, it rejects with "manifest invalid" because of uncompressed layer.
Steps to reproduce the issue: 1. 2. 3.
Describe the results you received:
Describe the results you expected:
Output of
rpm -q buildah
orapt list buildah
:Output of
buildah version
:Output of
podman version
if reporting apodman build
issue:*Output of `cat /etc/release`:**
Output of
uname -a
:Output of
cat /etc/containers/storage.conf
: