Skopeo producing different layer shas and sizes for the exact same docker image layer

[kaajoldhana]

Hello, I am using skopeo to compare the layers of 2 images. Both images are using ubuntu:jammy-20231128 as the base image. However, for the base image layer for image1 and image2, the sha of the layers are different and the size is different. All of the other layers in the image are the same. When comparing the layers with docker, all of the layers match.

Any ideas on why/how this could be happening? Both images are built with docker and are not using any additional compression besides the standard gzip.

skopeo inspect docker://ubuntu:jammy-20231128 -> "Layers": [ "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50" ],

skopeo inspect docker://<image1> -> "Layers": [ "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50", "sha256:cf61bcd72149b6f336e0cca5ad8ca43381dccebab8ef2aac3013e993a82fa466", "sha256:24af9922b1dd9c790eed664f4d6937c925c723ffbff4e2dd94c785f8f7c52a88", "sha256:d96efba7458332436f40a85187abbb430ed507086cf869d637aff5e7e8b41777", ] DockerHistory { "Comment": "", "Created": "2023-12-01T07:49:50.128319304Z", "CreatedBy": "/bin/sh -c #(nop) ADD file:36d444e2cede3abe58191dcf28890b874c0908f5259bf7e8855338555701c4c5 in / ", "Id": "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50", "Size": 29547417, "Tags": [] },

skopeo inspect docker://<image2> -> "Layers": [ "sha256:cbe3537751ce03ea42788c2fbe2d5d336180dc2e20472c8cdba8b3224191d101", "sha256:cf61bcd72149b6f336e0cca5ad8ca43381dccebab8ef2aac3013e993a82fa466", "sha256:24af9922b1dd9c790eed664f4d6937c925c723ffbff4e2dd94c785f8f7c52a88", "sha256:d96efba7458332436f40a85187abbb430ed507086cf869d637aff5e7e8b41777", ] DockerHistory { "Comment": "", "Created": "2023-12-01T07:49:50.128319304Z", "CreatedBy": "/bin/sh -c #(nop) ADD file:36d444e2cede3abe58191dcf28890b874c0908f5259bf7e8855338555701c4c5 in / ", "Id": "sha256:cbe3537751ce03ea42788c2fbe2d5d336180dc2e20472c8cdba8b3224191d101", "Size": 30446322, "Tags": [] },

[mtrmac]

Thanks for reaching out.

Please provide actual steps to reproduce; it’s completely unclear to me what is the relationship between those images.

Vaguely guessing: compressing the same uncompressed data is, in general, never guaranteed to consistently produce the same output, and the output can change at any time for any reason — different implementation, different version of the same implementation, different random ASLR choices, intentionally nondeterministic behavior.

[kaajoldhana]

@mtrmac Unfortunately, the images I provided above were from private registries so I won't be able to provide examples for 2 images that have a different top layer and the rest of the layers are the same but I have 2 random images that I was able to find that are using the same Ubuntu distribution but have different layer shas from Skopeo.

Image1: ghcr.io/rse-ops/intel-ubuntu-22.04:intel-2022.2.0 Image2: nvcr.io/nvidia/pytorch:23.12-py3

Commands: docker inspect ghcr.io/rse-ops/intel-ubuntu-22.04:intel-2022.2.0 "Layers": [ "sha256:8ceb9643fb36a8ac65882c07e7b2fff9fd117673d6784221a83d3ad076a9733e", "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", "sha256:4ea6576cad91981b8c77f66863b1c27557bc30508ca3e1e385770a1f7854b410", "sha256:d5915a6758b134610c91413c80ac15cb9e79484d37f9c7fd6e5b8f3f5f09a72e", "sha256:afac3162cdf155d0ddf62e0c3f207f755e468a8c142e143ae6de5fbde1ca815e", "sha256:acd7eab213c250d28eecda2b066c3f9e10324a52c2c8e4a413a3d07895a71bc8" ]

skopeo inspect docker://ghcr.io/rse-ops/intel-ubuntu-22.04:intel-2022.2.0 "Layers": [ "sha256:cbe3537751ce03ea42788c2fbe2d5d336180dc2e20472c8cdba8b3224191d101", "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1", "sha256:aebee794f307e5081f449930584a1ae17db8e1e1a3eeb347ef6f81a9992cba6c", "sha256:1932b123659fd395a2ca730f0fd6d3ba9b6f58419e0de3745c1fddac51ebc0c8", "sha256:d353affc45d265f88b281ef6a91c7df725b63f8f3808ec24920fabc97e1c7345", "sha256:35181f95df660412abfa15109569d6e19b868b152b42dbc67f375166627d96ae" ], docker inspect nvcr.io/nvidia/pytorch:23.12-py3 "Layers": [ "sha256:8ceb9643fb36a8ac65882c07e7b2fff9fd117673d6784221a83d3ad076a9733e", "sha256:b5724c8b6acf9860372dcfe573550e21990ba7de807a7810a02d49349e684556", "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", "sha256:c5c5573a8f787f8b6f71ee8845a1b1635ebb8c5e0d968bb16f3797b5420850d3", "sha256:81548b127d5e34a2dff0311f92ca3788f20d2a5a65255c40183fcde64d4db7d6", "sha256:bb668f86ed90962ce0481ddd48bb907dd86da79b705c81f0e649f47c0d85a35e", "sha256:2a1ce8564d505cdbfeabe384dd9c6ed5bbd086937aa5e12944effa3aa9bab731", "sha256:46cab564457586fdf6c2a599499edb00f6a3d004dcced4e5f4db0e4199252ff3", "sha256:4eae81cdb8a08f1be18623cb3ea8e2a8b4c1cdd2ece62d5c632d46bc44b27a64", "sha256:9a1417c6f900b15d700e7040ad6a0056178e9ae829dfef9d216d14e213d0c52a", "sha256:920806e544250cf4c73ea295b4e4863b03cc7aa29d3cdfbd4ae36f258960d081", "sha256:f34599c4bdc911393f8e5ea8662e7239f748b85d473411875ddebe82126cc9fd", ..... ], skopeo inspect docker://nvcr.io/nvidia/pytorch:23.12-py3` "Layers": [ "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50", "sha256:9075b8c4201a4d6dea7bbcbd610225804a66f25d51aaddb1a43e61dc105ff23d", "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1", "sha256:8847e45c417e982fe294a928fde99ace98327fea8338bf9b5a212090217fc655", "sha256:8a98aa4df5f4649df9ebb59b40df8761bc988448a81a3deafd0c42f915f4f521", "sha256:ebf2a8efd95be62a38b543798221c98a1bfcb91988640c031dd00c648c24df92", "sha256:cfc7cea7e0d5d1a44cae87fac92496ff8900e7a17ffbf87727c7fb2125f309c3", "sha256:eb738c4cfff7cf61cb24b239b9fb0e26de4627e909e20d9d173474f99d50fa7d", "sha256:51f661f07a8d57d9d766a456052e7777ea5ad8e20e3326073da9a628d38194b6", .... ]

I'm expecting that if the image layer shas when inspected using docker is the same that the image layer sha when inspecting using skopeo would also be the same.

Thank you for your help!

[mtrmac]

That’s just not how that works. “The same content” can have many different compressed representations = on-registry digests.

If you want to compare images to confirm equality, the layers need to be decompressed first. Or, alternatively, if you are in control of both images being compared, change your processes so that they are guaranteed to preserve the compressed representation (e.g. skopeo copy --preserve-digests).

[kaajoldhana]

@mtrmac are you aware of any differences between using

docker pull + docker tag + docker push

vs

skopeo copy --preserve-digests docker:// docker://

both image1 and image2 are already available in a registry and could be multi-arch (so --override-arch/--override-platform would also be used with skopeo copy)

appreciate your help with this!

[mtrmac]

@kaajoldhana What are you actually trying to do?

This is an issue tracker, not a place for an open-ended conversation.

[kaajoldhana]

@mtrmac I understand. Is there a better place to have a discussion?

I am looking to use skopeo to compare image layers between 2 images hosted in different registries. However, it seems like if docker is being used to push to the different registry, the layers are not guaranteed to match.

I am looking to understand how skopeo works and if there are any drawbacks to using skopeo over docker for the flow I described above

[mtrmac]

To “compare”, first decide whether you care about the compressed or the uncompressed representation; then compare accordingly.

I don’t think there is any downside to using skopeo copy --preserve-digests over docker pull + docker push — if you are in control over the workflow and you can guarantee that the other thing will not be used.

[kaajoldhana]

got it!

Thank you so much!