konrad-ohms
commented
3 years ago Just guessing, but might that be related in some kind? https://github.com/containers/buildah/issues/1589#issuecomment-504504999
Closed konrad-ohms closed 3 years ago
konrad-ohms
commented
3 years ago Just guessing, but might that be related in some kind? https://github.com/containers/buildah/issues/1589#issuecomment-504504999
konrad-ohms
commented
3 years ago It seems like I can workaround the problem by just pushing the OCI format to a local directory and using skopeo to push it as v2s2 to a registry. That seem to trigger a convert via skopeo and fixes the compression of layers.
This is my test script:
#!/bin/bash
set -xe
echo "Creating working container"
CTR=$(buildah from registry.access.redhat.com/ubi8-micro)
echo "Writing file into container fs"
buildah run "${CTR}" -- bash -c "echo hi>test.txt"
echo "Commit as oci image"
buildah commit --format oci "${CTR}" "local/validatedigest"
echo "Remove working container"
buildah rm "${CTR}"
echo "Ensure local fs is clean"
rm -rf images
echo "Push oci image to local fs via buildah"
buildah push --format oci "local/validatedigest" "oci:/$(pwd)/images:validatedigest:latest"
echo "Push local oci image as v2s2 to Artifactory via skopeo -> convert"
skopeo copy --format v2s2 "oci:/$(pwd)/images:validatedigest:latest" "docker://myartifactory.com/konrad/validatedigest"
echo "Show image details with skopeo inspect"
skopeo inspect --raw "docker://myartifactory.com/konrad/validatedigest" | jq "."Output:
$ ./checkDigestFix.sh
+ echo 'Creating working container'
Creating working container
++ buildah from registry.access.redhat.com/ubi8-micro
+ CTR=ubi8-micro-working-container-12
+ echo 'Writing file into container fs'
Writing file into container fs
+ buildah run ubi8-micro-working-container-12 -- bash -c 'echo hi>test.txt'
+ echo 'Commit as oci image'
Commit as oci image
+ buildah commit --format oci ubi8-micro-working-container-12 local/validatedigest
Getting image source signatures
Copying blob 5f70bf18a086 skipped: already exists
Copying blob a80a43bc8da6 skipped: already exists
Copying blob 99c72e14636b done
Copying config b07b72359f done
Writing manifest to image destination
Storing signatures
b07b72359fb256aab08fb4bbf139bd6868e8c4fe614f0eb9276eb638061737e2
+ echo 'Remove working container'
Remove working container
+ buildah rm ubi8-micro-working-container-12
9a9bd638481f24f95cbc91b3717fe6f82b62c826177e3e8fcc52af82a550de6e
+ echo 'Ensure local fs is clean'
Ensure local fs is clean
+ rm -rf images
+ echo 'Push oci image to local fs via buildah'
Push oci image to local fs via buildah
++ pwd
+ buildah push --format oci local/validatedigest oci://home/xxx/images:validatedigest:latest
Getting image source signatures
Copying blob 5f70bf18a086 done
Copying blob a80a43bc8da6 done
Copying blob 99c72e14636b done
Copying config b07b72359f done
Writing manifest to image destination
Storing signatures
+ echo 'Push local oci image as v2s2 to Artifactory via skopeo -> convert'
Push local oci image as v2s2 to Artifactory via skopeo -> convert
++ pwd
+ skopeo copy --format v2s2 oci://home/xxx/images:validatedigest:latest docker://myartifactory.com/konrad/validatedigest
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 4b9a13ae5bdb skipped: already exists
Copying blob dba4f7d26687 done
Copying config b07b72359f done
Writing manifest to image destination
Storing signatures
+ echo 'Show image details with skopeo inspect'
Show image details with skopeo inspect
+ skopeo inspect --raw docker://myartifactory.com/konrad/validatedigest
+ jq .
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1552,
"digest": "sha256:b07b72359fb256aab08fb4bbf139bd6868e8c4fe614f0eb9276eb638061737e2"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 233,
"digest": "sha256:dba4f7d26687c307706d5615fd4d5a21cc6654b6c4e4eba714704e7d177aeeb4"
}
]
}In that example the first layer is compressed again:
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
rhatdan
commented
3 years ago @vrothberg @nalind @mtrmac PTAL
mtrmac
commented
3 years ago but if the first image layer is uncompressed and pushed to Artifactory with buildah, I cannot promote it to a cloud registry without changing its digest, as skopeo compresses that layer automatically and thereby changes the size of the image.
If talking about builds, in general I’d say this is just an unreasonable assumption. Pulling an image to local storage, and pushing it back, whether just a podman pull+podman push, or using buildah, is not in general going to reconstruct layers; the compressed form is lost, and re-compression can always produce a different binary form of the original layer.
As implemented now, the tooling tries fairly hard to avoid layer uploads, so if it knows from previous history that the registry already contains some version of that layer, however it has been compressed, it is going to try to reuse it instead of pushing it again; which version exactly is used is deterministic but not an end-user promise and might change over version updates, so it’s not possible to build a reliable process around that. (And as a corner case of this, if the uncompressed version of that layer exists on the registry for some reason, the pushed image is (with the current implementation, not as a promise) going to refer to that uncompressed version, instead of uploading a compressed one.)
In this case OCI workaround probably avoids the “uncompressed version found on the registry” case, maybe bypasses some other part of the caching mechanism, and this happens to work with the cache state and registry as is currently, but that could easily break again if the registry contained somewhat different blobs or the cache were a bit different, and the oci:→docker:// copy could just as easily decide to use some differently-compressed version that already exists on the registry.
If we completely ignore the build part, and talk about a registry-to-registry copy of an already-fully-formed image, the tools do reuse blobs the same way by default, but it does make sense to ask for a publishing step that publishes exactly what the source was, and does not do any blob substitution, even at the cost of extra layer uploads.
Right now, skopeo copy runs in such a no-changes-to-the-image mode if the image is signed (and --remove-signatures is not specified), or if the the image destination specifies a digest. So, I’d expect skopeo copy docker://$source@sha256:$digest docker://$dest@sha256:$digest && skopeo copy docker://$dest@sha256:$digest docker://$dest:$tag (with the second part creating the desired tag in a somewhat expensive way) to work as expected for unsigned images (but I didn’t test it right now).
That could almost certainly be made easier, by adding a skopeo copy / c/image/copy.Options option to not modify the manifest under any circumstances, even for unsigned images copied to a tag.
It’s still quite possible that there’s a bug of some sort that causes the uncompressed upload for a bad reason, and a log with debugging enabled might help diagnose that. (But fixing that bug would not result in a reliable guarantee about blob digests, because reuse can still happen; see above.)
vrothberg
commented
3 years ago Let's check what we can do to resolve the immediate issue. Layer 5f70bf18a086 already exists in uncompressed form on the registry, so Buildah won't upload (and compress!) it.
Copying blob c68ab91c9118 skipped: already exists
Copying blob 5f70bf18a086 skipped: already exists
[...]
"layers": [
{
"mediaType": "application/vnd.oci.image.layer.v1.tar",
"digest": "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
"size": 1024
},
{
"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
"digest": "sha256:c68ab91c9118a78769d2559a3bb1091dc12a653673ad610f4995fcbd3cf68159",
"size": 13548478
},What we could do is to use skopeo copy --dest-compress-format=gzip instead of buildah push. That would enforce compressing the layers and would ultimately preserve the image digests when being copied any further from the registry to another.
mtrmac
commented
3 years ago What we could do is to use
skopeo copy --dest-compress-format=gzipinstead ofbuildah push. That would enforce compressing the layers
That doesn’t work AFAICS, (right now) layer reuse happens before a compression decision can be made. (And it’s still a futile endeavor to expect a buildah push to end up with exactly the same compressed digest as the last time. Maybe tomorrow will come a critical Go security update that changes the compression outcome.)
and would ultimately preserve the image digests when being copied any further from the registry to another.
I can’t see how that’s guaranteed: skopeo copy in the default mode is happy to avoid copies and reuse layers; whether the original is compressed or not, and how, plays only a very minor role in the layer reuse decision. If the toolchain is not explicitly preserving the original manifest digest, tinkering with what the current version happens to do in some specific situation is unlikely to lead to a reliable process.
I’m not at all opposed to a something like a copy.Options toggle to require a bit-identical copy, but I don’t think it makes sense for that to be the default/only behavior, and it’s not the default behavior of the current skopeo copy.
vrothberg
commented
3 years ago What we could do is to use
skopeo copy --dest-compress-format=gzipinstead ofbuildah push. That would enforce compressing the layersThat doesn’t work AFAICS, (right now) layer reuse happens before a compression decision can be made.
I was sure setting the compression would enforce copying all layers but you are right, it doesn't. Thanks for pointing that out.
(And it’s still a futile endeavor to expect a
buildah pushto end up with exactly the same compressed digest as the last time. Maybe tomorrow will come a critical Go security update that changes the compression outcome.)
That's correct but I am not suggesting to enable bit-identical compression over time. As you described above, that's not possible.
What I would like is a switch to enforce compression for the initial push such that subsequent copy operations of the same image won't alter the layers anymore (and hence preserve them).
mtrmac
commented
3 years ago What I would like is a switch to enforce compression for the initial push such that subsequent copy operations of the same image won't alter the layers anymore (and hence preserve them).
That’s not guaranteed either. If the copy can pair a source layer digest → DiffID → other layer known to exist on the destination, a plain skopeo copy of an unsigned image to a tag will substitute.
konrad-ohms
commented
3 years ago Thank you all for your quick help and detailed explanation. Okay, so you mean that the ubi8-micro image is most likely the problem itself as it has an uncompressed layer and uses signatures? So that would explain my observations, as the official image can be copied with unmodified digests as it copies signatures. Once I try to copy one of my unsigned images, it will detect that there is the uncompressed layer already in certain cases and the digest changes.
I was wondering why it worked in the past for us, but from your comments above, that was more or less a lucky coincidence that layers were not present on the target registry and not a skopeo feature we should have relied on.
If I understand your statements correctly, my workaround is not a reliable and can also break in certain conditions, I guess the best way to fix our current problem would be to sign the images in Artifactory before we promote them to IBM Cloud Registry to enforce the no-changes-to-the-image mode. @vrothberg I also tried to use --dest-compress-format=gzip and also --dest-compress-format=zstd without success, that would match the behavior as described by @mtrmac as the image layers were not pushed if they were already present before compression is applied.
I like both ideas, enforcing compression on initial pushes and adding a flag on skopeo to enforce the no-changes-to-the-image mode.
rhatdan
commented
3 years ago @jnovy @fatherlinux FYI
mtrmac
commented
3 years ago Okay, so you mean that the ubi8-micro image is most likely the problem itself as it has an uncompressed layer and uses signatures?
The ubi8-micro image, right now, doesn’t have an uncompressed layer. The most immediate cause is probably that the destination repo contains an uncompressed version of one of the layers, however it got there (intentionally or through a bug in some tooling); and that can happen again any time.
[It’s also a bit notable that the layer in question, 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 , is a compressed version of 1024 zero bytes, i.e. an “empty layer”, but compressed differently than what we consider the canonical representation with digest a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 . So even if the uncompressed version were not on the registry, it seems to me the build image probably would be uploaded using the a3ed… version instead of the original 4f4 version. The details are unimportant, the general principle is that pull+build+push does not guarantee to preserve the original compression format.]
If I understand your statements correctly, my workaround is not a reliable and can also break in certain conditions, I guess the best way to fix our current problem would be to sign the images in Artifactory before we promote them to IBM Cloud Registry
(Note that the signatures discussed are the “simple signing” format https://github.com/containers/image/blob/main/docs/containers-signature.5.md , not Notary / Docker content trust; Artifactory seems to support the latter, I’m not sure whether it supports the former.)
konrad-ohms
commented
3 years ago Thanks, that makes sense. Okay, I guess you are right according to Artifactory signature support, the signing command does succeed:
$ skopeo copy --all "dir:$(pwd)/images" "docker://${IMAGE_TO_SIGN}" --sign-by xxxBut I had trouble getting a host validating the signatures, so probably the support for that is missing on Artifactory at the moment:
$ podman pull myartifactory.com/myimage:mysignedtag
Trying to pull myartifactory.com/myimage:mysignedtag...
A signature was required, but no signature exists
Error: Source image rejected: A signature was required, but no signature existsSo I guess the only option remaining to fix the problem is to copy images by digests and then copy the entire manifest list again which should detect that layers are already present and should just use them.
Probably I should not use skopeo directly in that case, but maybe some wrapper script like this to transfer images:
#!/bin/bash
set -e
echo "========================================================"
echo "Copy images by digests"
echo "========================================================"
FULL_SOURCE_IMAGE="${1}"
FULL_TARGET_IMAGE="${2}"
if [ -z "${FULL_SOURCE_IMAGE}" ] || [ -z "${FULL_TARGET_IMAGE}" ]; then
echo "Usage: copyImagePreservingDigests.sh mysourceregistry.com/mynamespace/myimage:mytag mytargetregistry.com/mynamespace/myimage:mytag"
exit 1
fi
SOURCE_IMAGE="$(echo ${FULL_SOURCE_IMAGE} | awk '{split($0,a,":"); print a[1]}')"
TARGET_IMAGE="$(echo ${FULL_TARGET_IMAGE} | awk '{split($0,a,":"); print a[1]}')"
IMAGE_TAG="$(echo ${FULL_TARGET_IMAGE} | awk '{split($0,a,":"); print a[2]}')"
INSPECT_OUTPUT=$(skopeo inspect --raw "docker://${FULL_SOURCE_IMAGE}")
MEDIA_TYPE=$(echo "${INSPECT_OUTPUT}" | jq -r ".mediaType")
echo "MEDIA_TYPE=${MEDIA_TYPE}"
SOURCE_DIGEST=$(skopeo inspect "docker://${FULL_SOURCE_IMAGE}" | jq -r ".Digest")
if [ "${MEDIA_TYPE}" == "application/vnd.docker.distribution.manifest.list.v2+json" ]; then
echo "Source image is a manifest list"
echo "Manifest list digests: ${SOURCE_IMAGE}@${SOURCE_DIGEST}"
echo "Getting individual digests of image layers"
INDIVIDUAL_IMAGE_DIGESTS=$(skopeo inspect --raw "docker://${FULL_SOURCE_IMAGE}" | jq -r ".manifests[].digest")
elif [ "${MEDIA_TYPE}" == "application/vnd.docker.distribution.manifest.v2+json" ]; then
echo "Source image is a single image"
INDIVIDUAL_IMAGE_DIGESTS=${SOURCE_DIGEST}
else
echo "Unknown media type, digest copy not supported by script"
exit 1
fi
for CURRENT_DIGEST in ${INDIVIDUAL_IMAGE_DIGESTS}; do
echo "Copy ${SOURCE_IMAGE}@${CURRENT_DIGEST} to ${TARGET_IMAGE}@${CURRENT_DIGEST}"
skopeo copy "docker://${SOURCE_IMAGE}@${CURRENT_DIGEST}" "docker://${TARGET_IMAGE}@${CURRENT_DIGEST}"
echo "Source details"
skopeo inspect --raw "docker://${SOURCE_IMAGE}@${CURRENT_DIGEST}" | jq "."
echo "Target details"
skopeo inspect --raw "docker://${TARGET_IMAGE}@${CURRENT_DIGEST}" | jq "."
done
echo "Copy original image tag ${IMAGE_TAG}"
skopeo copy --all "docker://${FULL_SOURCE_IMAGE}" "docker://${FULL_TARGET_IMAGE}"
echo "Successfully copied image ${FULL_SOURCE_IMAGE} to ${FULL_TARGET_IMAGE}"
echo "Show image digests on source and target"
echo "${FULL_SOURCE_IMAGE}:"
skopeo inspect --raw "docker://${FULL_SOURCE_IMAGE}" | jq "."
echo ""
echo "${FULL_TARGET_IMAGE}:"
skopeo inspect --raw "docker://${FULL_TARGET_IMAGE}" | jq "."Output:
$ ./copyImagePreservingDigests.sh myartifactory.com/node:14.17.3-ubi8-micro-20210715063409 myartifactory.com/konrad/node:14.17.3-ubi8-micro-20210715063409
========================================================
Copy images by digests
========================================================
MEDIA_TYPE=application/vnd.docker.distribution.manifest.list.v2+json
Source image is a manifest list
Manifest list digests: myartifactory.com/node@sha256:3e8fe84f8c198bfde74e68ea768ba8504c321e991544e2e76d8fed533eb2bc80
Getting individual digests of image layers
Copy myartifactory.com/node@sha256:ea1ee61e6238275c2fa40b1f93f499d253ff9a2abb13cd1bd0a841b3d0c22a85 to myartifactory.com/konrad/node@sha256:ea1ee61e6238275c2fa40b1f93f499d253ff9a2abb13cd1bd0a841b3d0c22a85
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 338c2f6ba69d skipped: already exists
Copying blob 4b9a13ae5bdb [--------------------------------------] 0.0b / 0.0b
Copying config 9a88857b9a [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Source details
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1594,
"digest": "sha256:9a88857b9a6ff493f5f27f77e90325854616b99b6fb4af67b12453fe9643980d"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 41152987,
"digest": "sha256:338c2f6ba69dd1ad315a3beb49331ce9620583cc4128b4333fed0671ac19bf5b"
}
]
}
Target details
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1594,
"digest": "sha256:9a88857b9a6ff493f5f27f77e90325854616b99b6fb4af67b12453fe9643980d"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 41152987,
"digest": "sha256:338c2f6ba69dd1ad315a3beb49331ce9620583cc4128b4333fed0671ac19bf5b"
}
]
}
Copy myartifactory.com/node@sha256:47cb35c9e055ad73935934149c519b32414c8fe399dc1c98debbfdd5a5b4d85d to myartifactory.com/konrad/node@sha256:47cb35c9e055ad73935934149c519b32414c8fe399dc1c98debbfdd5a5b4d85d
Getting image source signatures
Copying blob 973457c49e8c skipped: already exists
Copying blob 4ffdbd93fd85 skipped: already exists
Copying blob 4ca545ee6d5d [--------------------------------------] 0.0b / 0.0b
Copying config 88f81b1bac [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Source details
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1600,
"digest": "sha256:88f81b1baca32a38c8fe14bd713fd38fd5bb907e30c26bf70a2eb7dfac3df7a2"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 18496327,
"digest": "sha256:973457c49e8c0e76f2d4d95340d5c5d9c9248f1aea55f357e71cfd3792c8bb5b"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 44194019,
"digest": "sha256:4ffdbd93fd85b1656f026e698c5fb21a887ea26fddee0babe54f4d51d782913b"
}
]
}
Target details
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1600,
"digest": "sha256:88f81b1baca32a38c8fe14bd713fd38fd5bb907e30c26bf70a2eb7dfac3df7a2"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 18496327,
"digest": "sha256:973457c49e8c0e76f2d4d95340d5c5d9c9248f1aea55f357e71cfd3792c8bb5b"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 44194019,
"digest": "sha256:4ffdbd93fd85b1656f026e698c5fb21a887ea26fddee0babe54f4d51d782913b"
}
]
}
Copy myartifactory.com/node@sha256:0f253d34974ec31532008d68e3471484ade8994200bd7f90efb57dde53357e0b to myartifactory.com/konrad/node@sha256:0f253d34974ec31532008d68e3471484ade8994200bd7f90efb57dde53357e0b
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob a2d8663583d9 skipped: already exists
Copying blob a55c2c1c67e3 [--------------------------------------] 0.0b / 0.0b
Copying config dabc6aa927 [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Source details
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1593,
"digest": "sha256:dabc6aa92743173a1a780ab03a445bb66ce8b176d522ba025c90ddbf35450eb9"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 14395028,
"digest": "sha256:a2d8663583d9d057b0fa16622280f9520d84900272cb68190043a76140da0dfa"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 40941067,
"digest": "sha256:a55c2c1c67e3c3c558ff9c659b7c99eda759c667b94dfc1bbaaf4a7aaee279e7"
}
]
}
Target details
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1593,
"digest": "sha256:dabc6aa92743173a1a780ab03a445bb66ce8b176d522ba025c90ddbf35450eb9"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 14395028,
"digest": "sha256:a2d8663583d9d057b0fa16622280f9520d84900272cb68190043a76140da0dfa"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 40941067,
"digest": "sha256:a55c2c1c67e3c3c558ff9c659b7c99eda759c667b94dfc1bbaaf4a7aaee279e7"
}
]
}
Copy original image tag 14.17.3-ubi8-micro-20210715063409
Getting image list signatures
Copying 3 of 3 images in list
Copying image sha256:ea1ee61e6238275c2fa40b1f93f499d253ff9a2abb13cd1bd0a841b3d0c22a85 (1/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 4b9a13ae5bdb skipped: already exists
Copying blob 338c2f6ba69d [--------------------------------------] 0.0b / 0.0b
Copying config 9a88857b9a [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Copying image sha256:47cb35c9e055ad73935934149c519b32414c8fe399dc1c98debbfdd5a5b4d85d (2/3)
Getting image source signatures
Copying blob 4ffdbd93fd85 skipped: already exists
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 973457c49e8c [--------------------------------------] 0.0b / 0.0b
Copying config 88f81b1bac [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Copying image sha256:0f253d34974ec31532008d68e3471484ade8994200bd7f90efb57dde53357e0b (3/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob a55c2c1c67e3 skipped: already exists
Copying blob a2d8663583d9 [--------------------------------------] 0.0b / 0.0b
Copying config dabc6aa927 [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Writing manifest list to image destination
Storing list signatures
Successfully copied image myartifactory.com/node:14.17.3-ubi8-micro-20210715063409 to myartifactory.com/konrad/node:14.17.3-ubi8-micro-20210715063409
Show image digests on source and target
myartifactory.com/node:14.17.3-ubi8-micro-20210715063409:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:ea1ee61e6238275c2fa40b1f93f499d253ff9a2abb13cd1bd0a841b3d0c22a85",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:47cb35c9e055ad73935934149c519b32414c8fe399dc1c98debbfdd5a5b4d85d",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:0f253d34974ec31532008d68e3471484ade8994200bd7f90efb57dde53357e0b",
"platform": {
"architecture": "s390x",
"os": "linux"
}
}
]
}
myartifactory.com/konrad/node:14.17.3-ubi8-micro-20210715063409:
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:ea1ee61e6238275c2fa40b1f93f499d253ff9a2abb13cd1bd0a841b3d0c22a85",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:47cb35c9e055ad73935934149c519b32414c8fe399dc1c98debbfdd5a5b4d85d",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:0f253d34974ec31532008d68e3471484ade8994200bd7f90efb57dde53357e0b",
"platform": {
"architecture": "s390x",
"os": "linux"
}
}
]
}But I had trouble getting a host validating the signatures, so probably the support for that is missing on Artifactory at the moment:
You would probably need a sigstore, and all the associated infrastructure.
So I guess the only option remaining to fix the problem is to copy images by digests and then copy the entire manifest list again which should detect that layers are already present and should just use them.
It doesn’t work that way; the copy does not go looking for images on the destination (it can’t even if it wanted to, BTW). Copying the single-arch images one at a time makes no difference to the following manifest-list copy (apart from affecting layer-reuse caches, but that is a side effect not strong enough to allow building a process on top).
If anything works right now, the two-command copy I have suggested above should be it, and that should include multi-arch (with both copies using --all similarly). And yes, the second copy must be from the destination to the destination, not from the source to the destination, for the same reason — copies are independent.
konrad-ohms
commented
3 years ago Ah, okay, sorry, I assumed that it would only work on images directly not on manifest lists as it refers to multiple other images.
I updated the script like below, it's way less complicated in that case:
#!/bin/bash
set -e
echo "========================================================"
echo "Copy images by digests"
echo "========================================================"
FULL_SOURCE_IMAGE="${1}"
FULL_TARGET_IMAGE="${2}"
if [ -z "${FULL_SOURCE_IMAGE}" ] || [ -z "${FULL_TARGET_IMAGE}" ]; then
echo "Usage: copyImagePreservingDigests.sh docker://mysourceregistry.com/mynamespace/myimage:mytag docker://mytargetregistry.com/mynamespace/myimage:mytag"
exit 1
fi
SOURCE_IMAGE="$(echo ${FULL_SOURCE_IMAGE} | awk '{split($0,a,":"); print a[1]":"a[2]}')"
TARGET_IMAGE="$(echo ${FULL_TARGET_IMAGE} | awk '{split($0,a,":"); print a[1]":"a[2]}')"
echo "SOURCE_IMAGE=${SOURCE_IMAGE}"
echo "TARGET_IMAGE=${TARGET_IMAGE}"
SOURCE_DIGEST=$(skopeo inspect "${FULL_SOURCE_IMAGE}" | jq -r ".Digest")
echo "SOURCE_DIGEST=${SOURCE_DIGEST}"
echo "Copy images by digest ${SOURCE_IMAGE}@${SOURCE_DIGEST} to ${TARGET_IMAGE}@${SOURCE_DIGEST}"
skopeo copy --all "${SOURCE_IMAGE}@${SOURCE_DIGEST}" "${TARGET_IMAGE}@${SOURCE_DIGEST}"
echo "Restore named tag on target registry ${TARGET_IMAGE}@${SOURCE_DIGEST} -> ${FULL_TARGET_IMAGE}"
skopeo copy --all "${TARGET_IMAGE}@${SOURCE_DIGEST}" "${FULL_TARGET_IMAGE}"
TARGET_DIGEST=$(skopeo inspect "${FULL_TARGET_IMAGE}" | jq -r ".Digest")
EXIT_CODE=0
if [ "${SOURCE_DIGEST}" == "${TARGET_DIGEST}" ]; then
echo "Successfully copied images (digest match)"
else
echo "Digest mismatch ${SOURCE_DIGEST} != ${TARGET_DIGEST}"
EXIT_CODE=1
fi
echo "Show image digests on source and target"
echo "=========== ${FULL_SOURCE_IMAGE} ==========="
skopeo inspect --raw "${FULL_SOURCE_IMAGE}" | jq "."
echo ""
echo "=========== ${FULL_TARGET_IMAGE} ==========="
skopeo inspect --raw "${FULL_TARGET_IMAGE}" | jq "."
exit ${EXIT_CODE}Example output:
$ ./copyImagePreservingDigests.sh docker://myartifactory.com/node:14.17.3-ubi8-micro docker://icr.io/ko-dev/node:14.17.3-ubi8-micro
========================================================
Copy images by digests
========================================================
SOURCE_IMAGE=docker://myartifactory.com/node
TARGET_IMAGE=docker://icr.io/ko-dev/node
SOURCE_DIGEST=sha256:e7af22415804babe9757ac0f0e7777a84bf89dac36ea1b6e07406b8153a6863f
Copy images by digest docker://myartifactory.com/node@sha256:e7af22415804babe9757ac0f0e7777a84bf89dac36ea1b6e07406b8153a6863f to docker://icr.io/ko-dev/node@sha256:e7af22415804babe9757ac0f0e7777a84bf89dac36ea1b6e07406b8153a6863f
Getting image list signatures
Copying 3 of 3 images in list
Copying image sha256:ae01219cbb4fb6c57353bf3654bcb7af480ecdb876298f79d055dd2e927db845 (1/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 4b9a13ae5bdb skipped: already exists
Copying blob cbf88f263812 done
Copying config b55ab52db1 done
Writing manifest to image destination
Storing signatures
Copying image sha256:1b5f1d47919cfa97c233502751c127d89b6add45d8a0804bb8ab56a303f43d3d (2/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 8d98913d1f89 done
Copying blob 973457c49e8c skipped: already exists
Copying config 9b96c59164 done
Writing manifest to image destination
Storing signatures
Copying image sha256:56782b1d5493fc3d632e07c5b8129c5db8cc537234071715d3f00da918d33991 (3/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob a2d8663583d9 skipped: already exists
Copying blob 79d0385e58e2 done
Copying config 333ecc5907 done
Writing manifest to image destination
Storing signatures
Writing manifest list to image destination
Storing list signatures
Restore named tag on target registry docker://icr.io/ko-dev/node@sha256:e7af22415804babe9757ac0f0e7777a84bf89dac36ea1b6e07406b8153a6863f -> docker://icr.io/ko-dev/node:14.17.3-ubi8-micro
Getting image list signatures
Copying 3 of 3 images in list
Copying image sha256:ae01219cbb4fb6c57353bf3654bcb7af480ecdb876298f79d055dd2e927db845 (1/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 4b9a13ae5bdb skipped: already exists
Copying blob cbf88f263812 [--------------------------------------] 0.0b / 0.0b
Copying config b55ab52db1 [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Copying image sha256:1b5f1d47919cfa97c233502751c127d89b6add45d8a0804bb8ab56a303f43d3d (2/3)
Getting image source signatures
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 973457c49e8c skipped: already exists
Copying blob 8d98913d1f89 [--------------------------------------] 0.0b / 0.0b
Copying config 9b96c59164 [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Copying image sha256:56782b1d5493fc3d632e07c5b8129c5db8cc537234071715d3f00da918d33991 (3/3)
Getting image source signatures
Copying blob a2d8663583d9 skipped: already exists
Copying blob 4ca545ee6d5d skipped: already exists
Copying blob 79d0385e58e2 [--------------------------------------] 0.0b / 0.0b
Copying config 333ecc5907 [--------------------------------------] 0.0b / 1.6KiB
Writing manifest to image destination
Storing signatures
Writing manifest list to image destination
Storing list signatures
Successfully copied images (digest match)
Show image digests on source and target
=========== docker://myartifactory.com/node:14.17.3-ubi8-micro ===========
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:ae01219cbb4fb6c57353bf3654bcb7af480ecdb876298f79d055dd2e927db845",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:1b5f1d47919cfa97c233502751c127d89b6add45d8a0804bb8ab56a303f43d3d",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:56782b1d5493fc3d632e07c5b8129c5db8cc537234071715d3f00da918d33991",
"platform": {
"architecture": "s390x",
"os": "linux"
}
}
]
}
=========== docker://icr.io/ko-dev/node:14.17.3-ubi8-micro ===========
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:ae01219cbb4fb6c57353bf3654bcb7af480ecdb876298f79d055dd2e927db845",
"platform": {
"architecture": "amd64",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:1b5f1d47919cfa97c233502751c127d89b6add45d8a0804bb8ab56a303f43d3d",
"platform": {
"architecture": "ppc64le",
"os": "linux"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 753,
"digest": "sha256:56782b1d5493fc3d632e07c5b8129c5db8cc537234071715d3f00da918d33991",
"platform": {
"architecture": "s390x",
"os": "linux"
}
}
]
}Thank you, it looks quite good so far in my test environment, I am currently trying to get that tested, but I need to work with other people to validate that, as I do not have access to the production environment. I will confirm, if it worked in production as well.
mtrmac
commented
3 years ago Thanks for the confirmation. I have filed https://github.com/containers/skopeo/issues/1378 to track this as a Skopeo RFE.
konrad-ohms
commented
3 years ago Perfect, thank you very much :-)
Regarding your comment above:
Okay, so you mean that the ubi8-micro image is most likely the problem itself as it has an uncompressed layer and uses signatures?
The ubi8-micro image, right now, doesn’t have an uncompressed layer. The most immediate cause is probably that the destination repo contains an uncompressed version of one of the layers, however it got there (intentionally or through a bug in some tooling); and that can happen again any time.
[It’s also a bit notable that the layer in question, 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 , is a compressed version of 1024 zero bytes, i.e. an “empty layer”, but compressed differently than what we consider the canonical representation with digest a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 . So even if the uncompressed version were not on the registry, it seems to me the build image probably would be uploaded using the a3ed… version instead of the original 4f4 version. The details are unimportant, the general principle is that pull+build+push does not guarantee to preserve the original compression format.]
If I understand your statements correctly, my workaround is not a reliable and can also break in certain conditions, I guess the best way to fix our current problem would be to sign the images in Artifactory before we promote them to IBM Cloud Registry
(Note that the signatures discussed are the “simple signing” format https://github.com/containers/image/blob/main/docs/containers-signature.5.md , not Notary / Docker content trust; Artifactory seems to support the latter, I’m not sure whether it supports the former.)
I basically I am the only developer in our broader team which migrated image builds from docker to buildah yet and afterwards the promotion process failed only on the images I created via buildah. Therefore, even if we might have a workaround for the promotion, I am trying to understand what might have caused the problem and how buildah output might differ from docker, even if we create artifacts with --format docker during builds or --format v2s2 during pushes. I think the empty layer might be one potential problem source.
I tried to understand what exactly triggered the creation of that layer. I am wondering if that empty layer should be created, or is that a problem which could be addressed on buildah? I avoided to use any registry in my test once buildah changed the image so that we can exclude any copy/convert problems in that scenario.
The scenario is:
demo-emtpy-layer.sh
#!/bin/bash
# digest fetched by: skopeo inspect docker://registry.access.redhat.com/ubi8-micro:latest | jq -r ".Digest"
set -e
echo "===== Empty Layer Demo ====="
echo "-> Building image"
BASE_IMAGE="registry.access.redhat.com/ubi8-micro@sha256:8b932917a156acbb1fb71b137c02c048d32abbf258babd9596d5ef89dfe8d401"
CTR=$(buildah from "${BASE_IMAGE}")
buildah run "${CTR}" -- /bin/bash -c "echo hello > /tmp/helloWorld.txt"
rm -rf "$(pwd)/empty-layer"
echo "-> Committing image (format docker) and push to local dir"
buildah commit --format docker --rm "${CTR}" local/empty-layer:latest
buildah push local/empty-layer:latest "dir://$(pwd)/empty-layer"
echo "-> Ensure that the file was written to the image correctly:"
podman run --rm -it localhost/local/empty-layer:latest cat /tmp/helloWorld.txt
echo "-> Creating an additional container based on the former one"
CTR=$(buildah from local/empty-layer:latest)
buildah run "${CTR}" -- /bin/bash -c "echo bye > /tmp/byeWorld.txt"
rm -rf "$(pwd)/empty-layer2"
echo "-> Committing image (format docker) and push to local dir"
buildah commit --format docker --rm "${CTR}" local/empty-layer2:latest
buildah push local/empty-layer2:latest "dir://$(pwd)/empty-layer2"
echo "-> Ensure that the second file was written added on top of the first image correctly:"
podman run --rm -it localhost/local/empty-layer2:latest cat /tmp/helloWorld.txt
podman run --rm -it localhost/local/empty-layer2:latest cat /tmp/byeWorld.txt
echo ""
echo "######### Inspect original base image (docker://${BASE_IMAGE}) #########"
skopeo inspect "docker://${BASE_IMAGE}"
echo ""
echo "Raw inspect original base image (docker://${BASE_IMAGE})"
skopeo inspect --raw "docker://${BASE_IMAGE}"
echo ""
echo "Raw inspect amd64 image to see layers (docker://registry.access.redhat.com/ubi8-micro@sha256:d389fa0cfee2091b3dd75a446fb982714db67bfb0916369f48985923890e2597)"
skopeo inspect --raw "docker://registry.access.redhat.com/ubi8-micro@sha256:d389fa0cfee2091b3dd75a446fb982714db67bfb0916369f48985923890e2597"
echo ""
echo "######### Inspect local image (no registry involved) (dir://$(pwd)/empty-layer) #########"
skopeo inspect "dir://$(pwd)/empty-layer"
echo ""
echo "Raw inspect local image (no registry involved) (dir://$(pwd)/empty-layer)"
skopeo inspect --raw "dir://$(pwd)/empty-layer"
echo ""
echo "Prettify output"
skopeo inspect --raw "dir://$(pwd)/empty-layer" | jq "."
echo ""
echo "######### Inspect second local image (no registry involved) (dir://$(pwd)/empty-layer2) #########"
skopeo inspect "dir://$(pwd)/empty-layer2"
echo ""
echo "Raw inspect local image (no registry involved) (dir://$(pwd)/empty-layer2)"
echo ""
skopeo inspect --raw "dir://$(pwd)/empty-layer2"
echo ""
echo "Prettify output"
skopeo inspect --raw "dir://$(pwd)/empty-layer2" | jq "."
echo ""
echo "######### Tool version information #########"
buildah version
buildah --version
skopeo --version
cat /etc/os-releaseOutput:
$ ./demo-empty-layer.sh
===== Empty Layer Demo =====
-> Building image
-> Committing image (format docker) and push to local dir
Getting image source signatures
Copying blob 5f70bf18a086 skipped: already exists
Copying blob a80a43bc8da6 skipped: already exists
Copying blob 7f0b1c849deb done
Copying config 469dad5a22 done
Writing manifest to image destination
Storing signatures
469dad5a2240b6691ba5f677532c46e691c20c595cc1baa8d9b8ad4154acf33c
Getting image source signatures
Copying blob 5f70bf18a086 done
Copying blob a80a43bc8da6 done
Copying blob 7f0b1c849deb done
Copying config 469dad5a22 done
Writing manifest to image destination
Storing signatures
-> Ensure that the file was written to the image correctly:
hello
-> Creating an additional container based on the former one
-> Committing image (format docker) and push to local dir
Getting image source signatures
Copying blob 5f70bf18a086 skipped: already exists
Copying blob a80a43bc8da6 skipped: already exists
Copying blob 7f0b1c849deb skipped: already exists
Copying blob fbeb028200b8 done
Copying config e8d69b9409 done
Writing manifest to image destination
Storing signatures
e8d69b940976a771ffbc00b559ac431f2b9a2f45a2b5567e4ab1cee69025f456
Getting image source signatures
Copying blob 5f70bf18a086 done
Copying blob a80a43bc8da6 done
Copying blob 7f0b1c849deb done
Copying blob fbeb028200b8 done
Copying config e8d69b9409 done
Writing manifest to image destination
Storing signatures
-> Ensure that the second file was written added on top of the first image correctly:
hello
bye
######### Inspect original base image (docker://registry.access.redhat.com/ubi8-micro@sha256:8b932917a156acbb1fb71b137c02c048d32abbf258babd9596d5ef89dfe8d401) #########
{
"Name": "registry.access.redhat.com/ubi8-micro",
"Digest": "sha256:8b932917a156acbb1fb71b137c02c048d32abbf258babd9596d5ef89dfe8d401",
"RepoTags": [
"8.4",
"8.4-72-source",
"8.4-72",
"8.4-81",
"8.4-81-source",
"latest"
],
"Created": "2021-06-22T12:49:12.088047363Z",
"DockerVersion": "1.13.1",
"Labels": {
"architecture": "x86_64",
"build-date": "2021-06-22T12:48:21.257644",
"com.redhat.build-host": "cpt-1003.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component": "ubi8-micro-container",
"com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
"description": "Very small image which doesn't install the package manager.",
"distribution-scope": "public",
"io.k8s.description": "Very small image which doesn't install the package manager.",
"io.k8s.display-name": "Ubi8-micro",
"io.openshift.expose-services": "",
"maintainer": "Red Hat, Inc.",
"name": "ubi8/ubi-micro",
"release": "81",
"summary": "ubi8 micro image",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/ubi-micro/images/8.4-81",
"vcs-ref": "c20f4a2add7d519164f7cf64842bc9f024d225ab",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "8.4"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
"sha256:c68ab91c9118a78769d2559a3bb1091dc12a653673ad610f4995fcbd3cf68159"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
}
Raw inspect original base image (docker://registry.access.redhat.com/ubi8-micro@sha256:8b932917a156acbb1fb71b137c02c048d32abbf258babd9596d5ef89dfe8d401)
{
"manifests": [
{
"digest": "sha256:d389fa0cfee2091b3dd75a446fb982714db67bfb0916369f48985923890e2597",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 735
},
{
"digest": "sha256:ba33bc5070c4317f4b5aeb37cde8b871a6f96123c036a0fe90bd9fef9de0db27",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm64",
"os": "linux"
},
"size": 735
},
{
"digest": "sha256:80db66427bd388fa9b8b7318fc3b9d74b02d23d5508be4da7066a78f3e277ea3",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"size": 735
},
{
"digest": "sha256:5381d3c44575c3be1ed54b0ea43947a58ef142099db41069417d023bff5bcf2e",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "s390x",
"os": "linux"
},
"size": 735
}
],
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"schemaVersion": 2
}
Raw inspect amd64 image to see layers (docker://registry.access.redhat.com/ubi8-micro@sha256:d389fa0cfee2091b3dd75a446fb982714db67bfb0916369f48985923890e2597)
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 2061,
"digest": "sha256:22d40a7f47a93c3bb90f5e9116c636b3d4d155c5ebe78ed9a95ae8dbd6a89d66"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 32,
"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 13548478,
"digest": "sha256:c68ab91c9118a78769d2559a3bb1091dc12a653673ad610f4995fcbd3cf68159"
}
]
}
######### Inspect local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer) #########
{
"Digest": "sha256:9ddbbdc8c6ffb85a2890e952e75733a0caa25e21cf09377e254205452c1bcee0",
"RepoTags": [],
"Created": "2021-07-16T09:29:44.356067771Z",
"DockerVersion": "",
"Labels": {
"architecture": "x86_64",
"build-date": "2021-06-22T12:48:21.257644",
"com.redhat.build-host": "cpt-1003.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component": "ubi8-micro-container",
"com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
"description": "Very small image which doesn't install the package manager.",
"distribution-scope": "public",
"io.buildah.version": "1.19.8",
"io.k8s.description": "Very small image which doesn't install the package manager.",
"io.k8s.display-name": "Ubi8-micro",
"io.openshift.expose-services": "",
"maintainer": "Red Hat, Inc.",
"name": "ubi8/ubi-micro",
"release": "81",
"summary": "ubi8 micro image",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/ubi-micro/images/8.4-81",
"vcs-ref": "c20f4a2add7d519164f7cf64842bc9f024d225ab",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "8.4"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3",
"sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628",
"sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
}
Raw inspect local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer)
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":3178,"digest":"sha256:469dad5a2240b6691ba5f677532c46e691c20c595cc1baa8d9b8ad4154acf33c"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":42,"digest":"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":15104061,"digest":"sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":260,"digest":"sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"}]}
Prettify output
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 3178,
"digest": "sha256:469dad5a2240b6691ba5f677532c46e691c20c595cc1baa8d9b8ad4154acf33c"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 260,
"digest": "sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"
}
]
}
######### Inspect second local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer2) #########
{
"Digest": "sha256:a049c5c79f2403d36fb2ac89637827aa0ffaca26a32b0cd05f543d78b986735f",
"RepoTags": [],
"Created": "2021-07-16T09:29:48.598273271Z",
"DockerVersion": "",
"Labels": {
"architecture": "x86_64",
"build-date": "2021-06-22T12:48:21.257644",
"com.redhat.build-host": "cpt-1003.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component": "ubi8-micro-container",
"com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
"description": "Very small image which doesn't install the package manager.",
"distribution-scope": "public",
"io.buildah.version": "1.19.8",
"io.k8s.description": "Very small image which doesn't install the package manager.",
"io.k8s.display-name": "Ubi8-micro",
"io.openshift.expose-services": "",
"maintainer": "Red Hat, Inc.",
"name": "ubi8/ubi-micro",
"release": "81",
"summary": "ubi8 micro image",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/ubi-micro/images/8.4-81",
"vcs-ref": "c20f4a2add7d519164f7cf64842bc9f024d225ab",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "8.4"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3",
"sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628",
"sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce",
"sha256:1062df870a6e573633cd19c8b4fd6b3687b26969eb924472c6838c55ce96ff27"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
]
}
Raw inspect local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer2)
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":3320,"digest":"sha256:e8d69b940976a771ffbc00b559ac431f2b9a2f45a2b5567e4ab1cee69025f456"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":42,"digest":"sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":15104061,"digest":"sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":260,"digest":"sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":158,"digest":"sha256:1062df870a6e573633cd19c8b4fd6b3687b26969eb924472c6838c55ce96ff27"}]}
Prettify output
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 3320,
"digest": "sha256:e8d69b940976a771ffbc00b559ac431f2b9a2f45a2b5567e4ab1cee69025f456"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 260,
"digest": "sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 158,
"digest": "sha256:1062df870a6e573633cd19c8b4fd6b3687b26969eb924472c6838c55ce96ff27"
}
]
}
######### Tool version information #########
Version: 1.19.8
Go Version: go1.15.13
Image Spec: 1.0.1-dev
Runtime Spec: 1.0.2-dev
CNI Spec: 0.4.0
libcni Version:
image Version: 5.10.5
Git Commit:
Built: Thu Jan 1 01:00:00 1970
OS/Arch: linux/amd64
buildah version 1.19.8 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)
skopeo version 1.2.3-dev
NAME="Red Hat Enterprise Linux"
VERSION="8.4 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.4"I noticed the following behavior which I do not fully understand: 1) On the UBI repo, the raw inspect returns a prettified JSON response directly (with whitespaces), but once buildah creates an image the created manifest is returned by skopeo as a single line. Not sure what exactly is used in calculating a manifest list digest, but if the raw content of the manifest list is compared whitespace would matter, I do not know if registries are prettifying those JSONs or if any tooling would do that, but at least it is a change from the original docker lists.
$ echo '{ "hi": "test" }' | sha256sum
9632f12b274cff96fae1433933ebf52be6a2dd07673345c872c5864c518f3e36 -
$ echo '{ "hi": "test" }' | sha256sum
efac789a00034a219ade27fa78be8c21d8e96d8729cfbd83fb40f548a2c59ea1 -2) The UBI base image contained two layers, I extracted it locally, it contained the compressd JSON file containing the image meta data and the layer containing the rootfs. That was my expectation so far. The first image created by buildah has three layers, the b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce contains
.
├── etc
│ └── resolv.conf
├── run
│ └── secrets
└── tmp
└── helloWorld.txtI expected only the tmp/helloWorld.txt file, but the other changes are not a problem I think. The 4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628 archive contains the rootfs as expected. Additionally, there is the uncompressed json document describing the image in bf608176a622b9cef04002f72bbf8a5047b328b67fdf4951a40aafb81bfcecf0. The file 4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3 is the empty archive which is not expected.
What is even a bit more strange, if you create an additional image on top of the first one, you will not get a second empty layer, if it is a general problem with every buildah commit, I would have expected to see multiple empty layers. Or is it still a general problem with buildah commit but the second empty layer would replace the previous one as the layer gets the same checksum? If that would be the case, I would have expected to see the empty layer twice in the image manifest, but this is only mentioned once:
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},I do not know if that is a problem in terms of compatibility, but at least it looks like the docker produced format varies in that regards from the buildah produced format. The ubi8-micro image might be created by Docker, that could explain why no empty layer is present in it.
konrad-ohms
commented
3 years ago I just got confirmation, that the promotion of our images seem to have worked, thank you very much :-).
As the emtpy layer is a different problem, I am moving that to a new issue.
mtrmac
commented
3 years ago Therefore, even if we might have a workaround for the promotion, I am trying to understand what might have caused the problem and how buildah output might differ from docker, even if we create artifacts with
--format dockerduring builds or--format v2s2during pushes.
Basically this just returns us back to the top start of the conversation: the build process is not designed to guarantee preservation of the exact binary form of layers, so it doesn’t, or may not. That’s all there is to it.
(The OpenShift-used blob cache might affect how this exactly plays out, but even in that case the reuse-instead-of-upload behavior AFAICS means that preserving the original binary form isn’t guaranteed.)
I tried to understand what exactly triggered the creation of that layer.
The build process of ubi8-micro , as it currently exists, happened to create one. As far as I care that’s out of our control, just as if it is out of our control if the build process ubi8-micro happened to create 3 10-megabyte layers. Builds FROM some image then preserve the layer structure (without --squash).
(reordered)
What is even a bit more strange, if you create an additional image on top of the first one, you will not get a second empty layer, if it is a general problem with every
buildah commit, I would have expected to see multiple empty layers. Or is it still a general problem withbuildah commitbut the second empty layer would replace the previous one as the layer gets the same checksum? If that would be the case, I would have expected to see the empty layer twice in the image manifest, but this is only mentioned once:
Empty layers might be created in various situations, primarily in the schema1 representation. However in this case the particular empty layer in question exists just because it has existed in the original image, so asking why Buildah didn’t create a second one is mistaken just because Buildah didn’t create (on this machine) the first one either.
- On the UBI repo, the raw inspect returns a prettified JSON response directly (with whitespaces), but once buildah creates an image the created manifest is returned by skopeo as a single line. Not sure what exactly is used in calculating a manifest list digest, but if the raw content of the manifest list is compared whitespace would matter, I do not know if registries are prettifying those JSONs or if any tooling would do that, but at least it is a change from the original docker lists.
The specification says JSON, and any valid JSON representation is a fair game. Registries must preserve exactly the form that was uploaded, e.g. because the digests validate the exact form, but nothing requires newly-built images to use any specific representation.
- The UBI base image contained two layers, I extracted it locally, it contained the compressd JSON file containing the image meta data and the layer containing the rootfs. That was my expectation so far. The first image created by buildah has three layers, the b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce contains
. ├── etc │ └── resolv.conf ├── run │ └── secrets └── tmp └── helloWorld.txtI expected only the tmp/helloWorld.txt file, but the other changes are not a problem I think.
I don’t know, this might be unexpected, or fine if the two other files are just mount points. That’s for Buildah experts to say, though. (@nalind ?)
konrad-ohms
commented
3 years ago Okay, thanks, in that case I don't open up a new issue. Ok, so what kept me wondering is why it created the empty layer on my system and on build servers and the script seem to be reproducible. As soon as I base it on ubi8-minimal, the empty layer is no longer there (attaching output at the bottom).
Regarding
The build process of ubi8-micro, as it currently exists, happened to create one. .... Empty layers might be created in various situations, primarily in the schema1 representation. However in this case the particular empty layer in question exists just because it has existed in the original image, so asking why Buildah didn’t create a second one is mistaken just because Buildah didn’t create (on this machine) the first one either.
The original ubi8-micro image contained only those layers
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 32,
"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 13548478,
"digest": "sha256:c68ab91c9118a78769d2559a3bb1091dc12a653673ad610f4995fcbd3cf68159"
}
]4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -> contained metadata c68ab91c9118a78769d2559a3bb1091dc12a653673ad610f4995fcbd3cf68159 -> contained the rootfs
I do not understand what you mean by ubi8-micro created that layer, I think it was added on the first buildah build running a fs write + commit, but I might be missing something you mentioned. Is buildah producing schema1 if buildah commit --format docker is used? From the output I assumed that everything is using schema version 2.
The first buildah build produced those layers which basically change exsting layers and added an empty one (but as you mentioned that binary representation is not guaranteed to be kept):
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 260,
"digest": "sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"
}
]The second build created:
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42,
"digest": "sha256:4ca545ee6d5db5c1170386eeb39b2ffe3bd46e5d4a73a9acbebc805f19607eb3"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 15104061,
"digest": "sha256:4b9a13ae5bdbf2207df8c9690b19ebb7bf50d6942cf3a3c081bad1b77d664628"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 260,
"digest": "sha256:b13bcac827fd0437039836e9b1c975d39158b2c8464070b4c40f1a3bf8639dce"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 158,
"digest": "sha256:1062df870a6e573633cd19c8b4fd6b3687b26969eb924472c6838c55ce96ff27"
}
]But if the build process is not designed to guarantee preservation of the exact binary form of layers that is probably still okay, from the outside it looked like there was some kind of transformation between ubi8-micro base image to the first custom build which created the emtpy layer and from the first custom build to the second custom build only an additional layer was added without changing the previous once.
Thanks for clarifying the manifest json representation question, I feared different JSON representations might be a problem as we got corrupted digests on image transfers only for the manifest lists lately, but that might be due to transfer of individual container images by digest, but not transfering the image list by digest.
The ubi8-minimal version without empty layers:
#!/bin/bash
# digest fetched by: skopeo inspect docker://registry.access.redhat.com/ubi8-minimal:latest | jq -r ".Digest"
set -e
echo "===== Empty Layer Demo ====="
echo "-> Building image"
BASE_IMAGE="registry.access.redhat.com/ubi8-minimal@sha256:b6b0c30bb747dfacee216e5ae2ad02adb18920d8f744c04f29354278e19df2a9"
CTR=$(buildah from "${BASE_IMAGE}")
buildah run "${CTR}" -- /bin/bash -c "echo hello > /tmp/helloWorld.txt"
rm -rf "$(pwd)/empty-layer"
echo "-> Committing image (format docker) and push to local dir"
buildah commit --format docker --rm "${CTR}" local/empty-layer:latest
buildah push local/empty-layer:latest "dir://$(pwd)/empty-layer"
echo "-> Ensure that the file was written to the image correctly:"
podman run --rm -it localhost/local/empty-layer:latest cat /tmp/helloWorld.txt
echo "-> Creating an additional container based on the former one"
CTR=$(buildah from local/empty-layer:latest)
buildah run "${CTR}" -- /bin/bash -c "echo bye > /tmp/byeWorld.txt"
rm -rf "$(pwd)/empty-layer2"
echo "-> Committing image (format docker) and push to local dir"
buildah commit --format docker --rm "${CTR}" local/empty-layer2:latest
buildah push local/empty-layer2:latest "dir://$(pwd)/empty-layer2"
echo "-> Ensure that the second file was written added on top of the first image correctly:"
podman run --rm -it localhost/local/empty-layer2:latest cat /tmp/helloWorld.txt
podman run --rm -it localhost/local/empty-layer2:latest cat /tmp/byeWorld.txt
echo ""
echo "######### Inspect original base image (docker://${BASE_IMAGE}) #########"
skopeo inspect "docker://${BASE_IMAGE}"
echo ""
echo "Raw inspect original base image (docker://${BASE_IMAGE})"
skopeo inspect --raw "docker://${BASE_IMAGE}"
echo ""
echo "Raw inspect amd64 image to see layers (docker://${BASE_IMAGE})"
skopeo inspect --raw "docker://registry.access.redhat.com/ubi8-minimal@sha256:ad1de47449386ae0f6009c38d714cbcc218d6f8087ea2d45272b671baa5072af"
echo ""
echo "######### Inspect local image (no registry involved) (dir://$(pwd)/empty-layer) #########"
skopeo inspect "dir://$(pwd)/empty-layer"
echo ""
echo "Raw inspect local image (no registry involved) (dir://$(pwd)/empty-layer)"
skopeo inspect --raw "dir://$(pwd)/empty-layer"
echo ""
echo "Prettify output"
skopeo inspect --raw "dir://$(pwd)/empty-layer" | jq "."
echo ""
echo "######### Inspect second local image (no registry involved) (dir://$(pwd)/empty-layer2) #########"
skopeo inspect "dir://$(pwd)/empty-layer2"
echo ""
echo "Raw inspect local image (no registry involved) (dir://$(pwd)/empty-layer2)"
echo ""
skopeo inspect --raw "dir://$(pwd)/empty-layer2"
echo ""
echo "Prettify output"
skopeo inspect --raw "dir://$(pwd)/empty-layer2" | jq "."
echo ""
echo "######### Tool version information #########"
buildah version
buildah --version
skopeo --version
cat /etc/os-releaseOutput:
./demo-non-empty-layer.sh
===== Empty Layer Demo =====
-> Building image
-> Committing image (format docker) and push to local dir
Getting image source signatures
Copying blob bc7bdf0ec1b9 skipped: already exists
Copying blob d7ecef9dcc97 skipped: already exists
Copying blob da2aa0334272 done
Copying config b557bb4090 done
Writing manifest to image destination
Storing signatures
b557bb4090ddb88aa524fd8a9344c5ef43c245e76b1e7233679d98527eefab21
Getting image source signatures
Copying blob bc7bdf0ec1b9 done
Copying blob d7ecef9dcc97 done
Copying blob da2aa0334272 done
Copying config b557bb4090 done
Writing manifest to image destination
Storing signatures
-> Ensure that the file was written to the image correctly:
hello
-> Creating an additional container based on the former one
-> Committing image (format docker) and push to local dir
Getting image source signatures
Copying blob bc7bdf0ec1b9 skipped: already exists
Copying blob d7ecef9dcc97 skipped: already exists
Copying blob da2aa0334272 skipped: already exists
Copying blob a70727951a06 done
Copying config d03fd70bbf done
Writing manifest to image destination
Storing signatures
d03fd70bbfb8fd9350363ec2994ffeed2c366b20e0deb9976e493f4c9f008920
Getting image source signatures
Copying blob bc7bdf0ec1b9 done
Copying blob d7ecef9dcc97 done
Copying blob da2aa0334272 done
Copying blob a70727951a06 done
Copying config d03fd70bbf done
Writing manifest to image destination
Storing signatures
-> Ensure that the second file was written added on top of the first image correctly:
hello
bye
######### Inspect original base image (docker://registry.access.redhat.com/ubi8-minimal@sha256:b6b0c30bb747dfacee216e5ae2ad02adb18920d8f744c04f29354278e19df2a9) #########
{
"Name": "registry.access.redhat.com/ubi8-minimal",
"Digest": "sha256:b6b0c30bb747dfacee216e5ae2ad02adb18920d8f744c04f29354278e19df2a9",
"RepoTags": [
"8.4-200",
"8.1-409-source",
"8.3-230",
"8.0-213",
"8.1-398-source",
"8.2-301",
"8.3-230-source",
"8.3-298.1618432845",
"8.1",
"8.0",
"8.3",
"8.2",
"8.3-291",
"8.2-349",
"8.0-204",
"8.1-407-source",
"8.2-345",
"8.3-201-source",
"8.3-298",
"8.2-301.1593113563",
"8.2-349-source",
"8.0-127",
"8.2-267",
"8.2-345-source",
"8.3-298.1618432845-source",
"8.2-301.1593113563-source",
"8.2-267-source",
"8.4-205",
"8.4-200.1622548483",
"8.2-301.1592810506-source",
"8.4-200-source",
"8.2-339",
"8.1-279",
"8.3-291-source",
"8.3-201",
"8.4",
"8.4-200.1622548483-source",
"8.2-301-source",
"8.2-339-source",
"8.1-328",
"8.0-159",
"8.2-301.1592810506",
"8.1-398",
"8.1-409",
"8.1-407",
"8.4-205-source",
"8.0-131",
"latest"
],
"Created": "2021-06-22T13:04:49.322942Z",
"DockerVersion": "1.13.1",
"Labels": {
"architecture": "x86_64",
"build-date": "2021-06-22T13:04:30.956781",
"com.redhat.build-host": "cpt-1005.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component": "ubi8-minimal-container",
"com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
"description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
"distribution-scope": "public",
"io.k8s.description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
"io.k8s.display-name": "Red Hat Universal Base Image 8 Minimal",
"io.openshift.expose-services": "",
"io.openshift.tags": "minimal rhel8",
"maintainer": "Red Hat, Inc.",
"name": "ubi8-minimal",
"release": "205",
"summary": "Provides the latest release of the minimal Red Hat Universal Base Image 8.",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-205",
"vcs-ref": "7256039d3c371a38cf13906dcf5688c19700c73b",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "8.4"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:96965a3a84248c364364702c0fb90543e329f86044b3394f97701f25b516b9ee",
"sha256:4d0d850cd4adc37289686142206a183ccbd4e286765ce8fc9890539bbfd38827"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
]
}
Raw inspect original base image (docker://registry.access.redhat.com/ubi8-minimal@sha256:b6b0c30bb747dfacee216e5ae2ad02adb18920d8f744c04f29354278e19df2a9)
{
"manifests": [
{
"digest": "sha256:48a4bec3d1dec90b5dd5420bf7c41a5756b7fbe8b862546134fbe2caa607679f",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "amd64",
"os": "linux"
},
"size": 737
},
{
"digest": "sha256:ad1de47449386ae0f6009c38d714cbcc218d6f8087ea2d45272b671baa5072af",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "arm64",
"os": "linux"
},
"size": 737
},
{
"digest": "sha256:82b6decae91cf9cc9c3d65e264920e351850b2b3f4711b07d58d815fd4cdab3d",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "ppc64le",
"os": "linux"
},
"size": 737
},
{
"digest": "sha256:3410b9e870bd51bd71646a85beb9e7dd767275208ff33f52c3277c1877c9a3b9",
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"platform": {
"architecture": "s390x",
"os": "linux"
},
"size": 737
}
],
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"schemaVersion": 2
}
Raw inspect amd64 image to see layers (docker://registry.access.redhat.com/ubi8-minimal@sha256:b6b0c30bb747dfacee216e5ae2ad02adb18920d8f744c04f29354278e19df2a9)
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4279,
"digest": "sha256:1c06a69806ac806581803d928499fc6aa4fed9115f04d3c10b312b1844655c88"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 39050730,
"digest": "sha256:05e70a9e592a04d8af34919e12f7fbabd0ea14becaae2d272afc1927b667bf95"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1742,
"digest": "sha256:e6efe05d74ecacf322bbe8f9174e4416d33aff17897ccc97a988e90be016db5e"
}
]
}
######### Inspect local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer) #########
{
"Digest": "sha256:77e6436d8bfe72064137ce473c87654d2c66a72f66ed8c302253a2b0de45f2c8",
"RepoTags": [],
"Created": "2021-07-16T14:38:51.60010252Z",
"DockerVersion": "",
"Labels": {
"architecture": "x86_64",
"build-date": "2021-06-22T13:04:30.956781",
"com.redhat.build-host": "cpt-1005.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component": "ubi8-minimal-container",
"com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
"description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
"distribution-scope": "public",
"io.buildah.version": "1.19.8",
"io.k8s.description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
"io.k8s.display-name": "Red Hat Universal Base Image 8 Minimal",
"io.openshift.expose-services": "",
"io.openshift.tags": "minimal rhel8",
"maintainer": "Red Hat, Inc.",
"name": "ubi8-minimal",
"release": "205",
"summary": "Provides the latest release of the minimal Red Hat Universal Base Image 8.",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-205",
"vcs-ref": "7256039d3c371a38cf13906dcf5688c19700c73b",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "8.4"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c",
"sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0",
"sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
]
}
Raw inspect local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer)
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":4545,"digest":"sha256:b557bb4090ddb88aa524fd8a9344c5ef43c245e76b1e7233679d98527eefab21"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":42185562,"digest":"sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1817,"digest":"sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":209,"digest":"sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"}]}
Prettify output
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4545,
"digest": "sha256:b557bb4090ddb88aa524fd8a9344c5ef43c245e76b1e7233679d98527eefab21"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42185562,
"digest": "sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1817,
"digest": "sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 209,
"digest": "sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"
}
]
}
######### Inspect second local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer2) #########
{
"Digest": "sha256:c810e695bc5eb546d1896928b7e878d5ac0a7691edb0d782124cb3f95e150442",
"RepoTags": [],
"Created": "2021-07-16T14:38:57.296357488Z",
"DockerVersion": "",
"Labels": {
"architecture": "x86_64",
"build-date": "2021-06-22T13:04:30.956781",
"com.redhat.build-host": "cpt-1005.osbs.prod.upshift.rdu2.redhat.com",
"com.redhat.component": "ubi8-minimal-container",
"com.redhat.license_terms": "https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI",
"description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
"distribution-scope": "public",
"io.buildah.version": "1.19.8",
"io.k8s.description": "The Universal Base Image Minimal is a stripped down image that uses microdnf as a package manager. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.",
"io.k8s.display-name": "Red Hat Universal Base Image 8 Minimal",
"io.openshift.expose-services": "",
"io.openshift.tags": "minimal rhel8",
"maintainer": "Red Hat, Inc.",
"name": "ubi8-minimal",
"release": "205",
"summary": "Provides the latest release of the minimal Red Hat Universal Base Image 8.",
"url": "https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8-minimal/images/8.4-205",
"vcs-ref": "7256039d3c371a38cf13906dcf5688c19700c73b",
"vcs-type": "git",
"vendor": "Red Hat, Inc.",
"version": "8.4"
},
"Architecture": "amd64",
"Os": "linux",
"Layers": [
"sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c",
"sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0",
"sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5",
"sha256:3aeb3a9b9a3cea3a8bb7db33d7f7a17ef83e64a220a6eb7943ad180fde2395d5"
],
"Env": [
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"container=oci"
]
}
Raw inspect local image (no registry involved) (dir:///home/ohmsk/Downloads/skopeo/emtpy-layer/empty-layer2)
{"schemaVersion":2,"mediaType":"application/vnd.docker.distribution.manifest.v2+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":4688,"digest":"sha256:d03fd70bbfb8fd9350363ec2994ffeed2c366b20e0deb9976e493f4c9f008920"},"layers":[{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":42185562,"digest":"sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":1817,"digest":"sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":209,"digest":"sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"},{"mediaType":"application/vnd.docker.image.rootfs.diff.tar.gzip","size":158,"digest":"sha256:3aeb3a9b9a3cea3a8bb7db33d7f7a17ef83e64a220a6eb7943ad180fde2395d5"}]}
Prettify output
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 4688,
"digest": "sha256:d03fd70bbfb8fd9350363ec2994ffeed2c366b20e0deb9976e493f4c9f008920"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42185562,
"digest": "sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1817,
"digest": "sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 209,
"digest": "sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 158,
"digest": "sha256:3aeb3a9b9a3cea3a8bb7db33d7f7a17ef83e64a220a6eb7943ad180fde2395d5"
}
]
}
######### Tool version information #########
Version: 1.19.8
Go Version: go1.15.13
Image Spec: 1.0.1-dev
Runtime Spec: 1.0.2-dev
CNI Spec: 0.4.0
libcni Version:
image Version: 5.10.5
Git Commit:
Built: Thu Jan 1 01:00:00 1970
OS/Arch: linux/amd64
buildah version 1.19.8 (image-spec 1.0.1-dev, runtime-spec 1.0.2-dev)
skopeo version 1.2.3-dev
NAME="Red Hat Enterprise Linux"
VERSION="8.4 (Ootpa)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="8.4"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8"
REDHAT_BUGZILLA_PRODUCT_VERSION=8.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.4"
The original ubi8-minimal image contained two layers like the ubi8-micro one:
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 39050730,
"digest": "sha256:05e70a9e592a04d8af34919e12f7fbabd0ea14becaae2d272afc1927b667bf95"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1742,
"digest": "sha256:e6efe05d74ecacf322bbe8f9174e4416d33aff17897ccc97a988e90be016db5e"
}
]-> both layers contain files
But buildah did not create an empty layer after the image build on top of ubi8-minimal:
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42185562,
"digest": "sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1817,
"digest": "sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 209,
"digest": "sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"
}
]And added a new layer on the second custom build:
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 42185562,
"digest": "sha256:7782f100031b10d1ebfdf01b4500b237d4457cf15d55748b4beeea505372508c"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 1817,
"digest": "sha256:399e5d2c71d84493b24c09ff0036c9f79c36a001ee70d37fe004f778897966b0"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 209,
"digest": "sha256:bece370f09992563c6dfd84438f7cf3aace7ce385b5d3f9c90b6ca97d7dd68d5"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 158,
"digest": "sha256:3aeb3a9b9a3cea3a8bb7db33d7f7a17ef83e64a220a6eb7943ad180fde2395d5"
}
]From what I saw, it might be some kind of coincidence that on the ubi8-micro to custom build transition an empty layer is created as the transition from ubi8-minimal to the first custom build also restructured the layers.
mtrmac
commented
3 years ago 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 -> contained metadata
No, that’s an empty layer (a compressed version that decompresses to 1 KB of zero bytes).
The image config of that image is blob 22d40a7f47a93c3bb90f5e9116c636b3d4d155c5ebe78ed9a95ae8dbd6a89d66 .
And if ubi8-minimal (as opposed to ubi8-micro) doesn’t have such an empty layer, well, that’s up to whoever created the two images.
konrad-ohms
commented
3 years ago Ah, I see, okay thank you very much for all the details, you helped me out quite a bit :-)
Description The
buildah budandbuildah commitcommands will not compress the first image layer if using ubi8-micro images, but are using gzip compression by default on following layers.Containers can be created based on the produced images, nevertheless I faced problems when promoting them with skopeo between registries. Our build relies on image digests to ensure that versions did not change, but if the first image layer is uncompressed and pushed to Artifactory with buildah, I cannot promote it to a cloud registry without changing its digest, as skopeo compresses that layer automatically and thereby changes the size of the image.
Steps to reproduce the issue:
$ buildah bud -t local/buildah-debug-bud:latest . STEP 1: FROM registry.access.redhat.com/ubi8-micro:latest Getting image source signatures Checking if image destination supports signatures Copying blob c68ab91c9118 skipped: already exists
Copying blob 4f4fb700ef54 [--------------------------------------] 0.0b / 0.0b Copying config 22d40a7f47 done
Writing manifest to image destination Storing signatures STEP 2: RUN echo "test" > /test STEP 3: COMMIT local/buildah-debug-bud:latest Getting image source signatures Copying blob 5f70bf18a086 skipped: already exists
Copying blob a80a43bc8da6 skipped: already exists
Copying blob 1e02c05341cb done
Copying config 27ab642c64 done
Writing manifest to image destination Storing signatures --> 27ab642c64b 27ab642c64be864bff3da2a122647ea35368e550e6ff24f8d070d805aa5c5f19
Describe the results you received: When inspecting the images, buildah push seem not to have compressed the first layer when pushing it to Artifactory, but skopeo compressed the first layer during the copy process to the IBM Cloud Registry. This changes the image digest, which gets checked in our deployment process to ensure unchanged build artifacts.
Please note the first layer is using tar
and the following layers use tar + gizip:
The image digest is
sha256:27ab642c64be864bff3da2a122647ea35368e550e6ff24f8d070d805aa5c5f19on Artifactory.Inspecting it on IBM Cloud Registry shows the following results:
Please note, that the first layer got compressed now:
As the image size changed due to applied compression, the digest is also different, it is using
sha256:0a0b1a8de715a2f64bfe8d229910481acde8c5d3d515ec10ca9a877d702d0748after the copy step.I tried the same using
buildah commitinstead ofbuildah budand for--format ociand--format docker(during build) and--format ociand--format v2s2during the push, the first layer is not getting compressed in all scenarios.In my build pipeline the individual images are build on amd64, ppc64le and s390x and a manifest list is getting created and promoted. To not overload the example, I only considered a single image above as the problem is visible there as well.
Note: Podman does show that behavior as well:
Describe the results you expected: All buildah image layers should be compressed, also the first layer. Also, the image digests should not change while copying images between registries with skopeo.
Output of
rpm -q buildahorapt list buildah:Output of
buildah version:Output of
podman versionif reporting apodman buildissue:*Output of `cat /etc/release`:**
Output of
uname -a:Output of
cat /etc/containers/storage.conf: