Build tooling for containers appears to generate empty layers on USER directives

[iamasmith]

Brief summary

This is more of an FYI

Your build tooling appears to be adding an empty 32 byte layer each time it encounters a USER directive in the build.

At work the image analysis pipeline I created highlighted this as an anomaly, it may concern you or it may not - just presenting this information.

This example looks at latest-with-browser (two USER directive in the build stage) which derives from the latest image (release in Dockerfile). This shows two empty layers whilst the latest image (release) uses one USER directive in its production and shows one empty layer.

I base the assumption on the USER directive due to the proximity of the layers as I do not know the tool used in your pipeline. If the layer artifact was due to image being derived from one would suspect that the two identical layers would be separated by an intermediary containing content. In this case they they are not and the only distinguishing factor seems to be the 2 USER directives which should not create layers according to Docker documentation but I imagine some build systems might incorrectly do this so I suspect the build system does isolate empty layers on next layer production and eliminate them but not on the final layer. Without knowing the build tool I cannot advise further.

andrews@AndrewsiMac1307 k6test % skopeo copy --override-os linux --override-arch amd64 docker://grafana/k6:latest-with-browser dir://$(pwd)
Getting image source signatures
Copying blob 4c99ea95ff02 done  
Copying blob 4f4fb700ef54 done  
Copying blob 96526aa774ef done  
Copying blob 39b555892f68 done  
Copying blob 4f4fb700ef54 done  
Copying blob 0a19ebaf598e done  
Copying config 04ffb09db9 done  
Writing manifest to image destination

# The 4f4fb700ef54 (truncated) sha256 appears twice so inspecting the image manifest

andrews@AndrewsiMac1307 k6test % cat manifest.json 
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:04ffb09db9028874176aa02f900dd75debc7dc580470fd47a099ffe64540a36b",
    "size": 3111
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa",
      "size": 3401967
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4c99ea95ff023e2025d014b83df298b7c8a02299331b7886b1d27174a78e993b",
      "size": 285922
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:39b555892f689fd851845f0cdbfbc50773f069c7fc68722b9a4929c9d8565de6",
      "size": 24368490
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
      "size": 32
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1",
      "size": 32
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:0a19ebaf598e112b36576aa84b5ae90ea1eae35f41f1b6a6c0eb08553040faa9",
      "size": 242592170
    }
  ]
}%

We can see the image manifest carries two 32 byte files which is actually smaller than an empty tar gz file and appear to be an empty gzip file with a 0 timestamp

k6 version

latest-with-browser

OS

Linux

Docker version and image (if applicable)

No response

Steps to reproduce the problem

Issue is a build system report

Expected behaviour

No duplicate layers and possibly no empty layers

Actual behaviour

Empty layer present on 'latest' and duplicate empty layers on 'latest-with-browser'

[oleiade]

Hey @iamasmith, apologies for the delay, this went under my radar for far too long 🙇🏻

Thanks for bringing this to our attention and for your analysis. Something unexpected seems to be happening indeed on that front, and it's probably something we should evaluate.

My main question mark would be as to whether or not that's an issue per see, considering it's a very tiny overhead.

[iamasmith]

On GKE any duplicate layer included makes the image incompatible with image streaming until adopting 1.28 https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming

[iamasmith]

Reported on the 19th April there appears to be an additional issue noted with Image Streaming that again appears to be triggered under certain circumstances by duplicate layers.