Skopeo producing different layer shas and sizes for the exact same docker image layer

[kaajoldhana]

Hello, I am using skopeo to compare the layers of 2 images. Both images are using ubuntu:jammy-20231128 as the base image. However, for the base image layer for image1 and image2, the sha of the layers are different and the size is different. All of the other layers in the image are the same. When comparing the layers with docker, all of the layers match.

Any ideas on why/how this could be happening? Both images are built with docker and are not using any additional compression besides the standard gzip.

skopeo inspect docker://ubuntu:jammy-20231128 -> "Layers": [ "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50" ],

skopeo inspect docker://<image1> -> "Layers": [ "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50", "sha256:cf61bcd72149b6f336e0cca5ad8ca43381dccebab8ef2aac3013e993a82fa466", "sha256:24af9922b1dd9c790eed664f4d6937c925c723ffbff4e2dd94c785f8f7c52a88", "sha256:d96efba7458332436f40a85187abbb430ed507086cf869d637aff5e7e8b41777", ] DockerHistory { "Comment": "", "Created": "2023-12-01T07:49:50.128319304Z", "CreatedBy": "/bin/sh -c #(nop) ADD file:36d444e2cede3abe58191dcf28890b874c0908f5259bf7e8855338555701c4c5 in / ", "Id": "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50", "Size": 29547417, "Tags": [] },

skopeo inspect docker://<image2> -> "Layers": [ "sha256:cbe3537751ce03ea42788c2fbe2d5d336180dc2e20472c8cdba8b3224191d101", "sha256:cf61bcd72149b6f336e0cca5ad8ca43381dccebab8ef2aac3013e993a82fa466", "sha256:24af9922b1dd9c790eed664f4d6937c925c723ffbff4e2dd94c785f8f7c52a88", "sha256:d96efba7458332436f40a85187abbb430ed507086cf869d637aff5e7e8b41777", ] DockerHistory { "Comment": "", "Created": "2023-12-01T07:49:50.128319304Z", "CreatedBy": "/bin/sh -c #(nop) ADD file:36d444e2cede3abe58191dcf28890b874c0908f5259bf7e8855338555701c4c5 in / ", "Id": "sha256:cbe3537751ce03ea42788c2fbe2d5d336180dc2e20472c8cdba8b3224191d101", "Size": 30446322, "Tags": [] },

[mtrmac]

Thanks for reaching out.

Please provide actual steps to reproduce; it’s completely unclear to me what is the relationship between those images.

Vaguely guessing: compressing the same uncompressed data is, in general, never guaranteed to consistently produce the same output, and the output can change at any time for any reason — different implementation, different version of the same implementation, different random ASLR choices, intentionally nondeterministic behavior.

[kaajoldhana]

@mtrmac Unfortunately, the images I provided above were from private registries so I won't be able to provide examples for 2 images that have a different top layer and the rest of the layers are the same but I have 2 random images that I was able to find that are using the same Ubuntu distribution but have different layer shas from Skopeo.

Image1: ghcr.io/rse-ops/intel-ubuntu-22.04:intel-2022.2.0 Image2: nvcr.io/nvidia/pytorch:23.12-py3

Commands: docker inspect ghcr.io/rse-ops/intel-ubuntu-22.04:intel-2022.2.0 "Layers": [ "sha256:8ceb9643fb36a8ac65882c07e7b2fff9fd117673d6784221a83d3ad076a9733e", "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", "sha256:4ea6576cad91981b8c77f66863b1c27557bc30508ca3e1e385770a1f7854b410", "sha256:d5915a6758b134610c91413c80ac15cb9e79484d37f9c7fd6e5b8f3f5f09a72e", "sha256:afac3162cdf155d0ddf62e0c3f207f755e468a8c142e143ae6de5fbde1ca815e", "sha256:acd7eab213c250d28eecda2b066c3f9e10324a52c2c8e4a413a3d07895a71bc8" ]

skopeo inspect docker://ghcr.io/rse-ops/intel-ubuntu-22.04:intel-2022.2.0 "Layers": [ "sha256:cbe3537751ce03ea42788c2fbe2d5d336180dc2e20472c8cdba8b3224191d101", "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1", "sha256:aebee794f307e5081f449930584a1ae17db8e1e1a3eeb347ef6f81a9992cba6c", "sha256:1932b123659fd395a2ca730f0fd6d3ba9b6f58419e0de3745c1fddac51ebc0c8", "sha256:d353affc45d265f88b281ef6a91c7df725b63f8f3808ec24920fabc97e1c7345", "sha256:35181f95df660412abfa15109569d6e19b868b152b42dbc67f375166627d96ae" ], docker inspect nvcr.io/nvidia/pytorch:23.12-py3 "Layers": [ "sha256:8ceb9643fb36a8ac65882c07e7b2fff9fd117673d6784221a83d3ad076a9733e", "sha256:b5724c8b6acf9860372dcfe573550e21990ba7de807a7810a02d49349e684556", "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef", "sha256:c5c5573a8f787f8b6f71ee8845a1b1635ebb8c5e0d968bb16f3797b5420850d3", "sha256:81548b127d5e34a2dff0311f92ca3788f20d2a5a65255c40183fcde64d4db7d6", "sha256:bb668f86ed90962ce0481ddd48bb907dd86da79b705c81f0e649f47c0d85a35e", "sha256:2a1ce8564d505cdbfeabe384dd9c6ed5bbd086937aa5e12944effa3aa9bab731", "sha256:46cab564457586fdf6c2a599499edb00f6a3d004dcced4e5f4db0e4199252ff3", "sha256:4eae81cdb8a08f1be18623cb3ea8e2a8b4c1cdd2ece62d5c632d46bc44b27a64", "sha256:9a1417c6f900b15d700e7040ad6a0056178e9ae829dfef9d216d14e213d0c52a", "sha256:920806e544250cf4c73ea295b4e4863b03cc7aa29d3cdfbd4ae36f258960d081", "sha256:f34599c4bdc911393f8e5ea8662e7239f748b85d473411875ddebe82126cc9fd", ..... ], skopeo inspect docker://nvcr.io/nvidia/pytorch:23.12-py3` "Layers": [ "sha256:5e8117c0bd28aecad06f7e76d4d3b64734d59c1a0a44541d18060cd8fba30c50", "sha256:9075b8c4201a4d6dea7bbcbd610225804a66f25d51aaddb1a43e61dc105ff23d", "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1", "sha256:8847e45c417e982fe294a928fde99ace98327fea8338bf9b5a212090217fc655", "sha256:8a98aa4df5f4649df9ebb59b40df8761bc988448a81a3deafd0c42f915f4f521", "sha256:ebf2a8efd95be62a38b543798221c98a1bfcb91988640c031dd00c648c24df92", "sha256:cfc7cea7e0d5d1a44cae87fac92496ff8900e7a17ffbf87727c7fb2125f309c3", "sha256:eb738c4cfff7cf61cb24b239b9fb0e26de4627e909e20d9d173474f99d50fa7d", "sha256:51f661f07a8d57d9d766a456052e7777ea5ad8e20e3326073da9a628d38194b6", .... ]

I'm expecting that if the image layer shas when inspected using docker is the same that the image layer sha when inspecting using skopeo would also be the same.

Thank you for your help!