podman build does not use parent image's layers

ibotty commented 1 year ago

Issue Description

Creating an image with base image registry.access.redhat.com/ubi9 does not use the parent image's layers sha256 digest.

Steps to reproduce the issue

create simple image

cat <<EOF > Containerfile
FROM registry.access.redhat.com/ubi9
RUN touch /a
EOF
podman build . -t localhost/test:latest --pull

inspect layers

skopeo inspect containers-storage:localhost/test:latest | jq .Layers

note that the parent image's layers are not included

skopeo inspect registry.access.redhat.com/ubi9 | jq .Layers

Describe the results you received

With

> skopeo inspect docker://registry.access.redhat.com/ubi9:latest| jq .Layers
[
  "sha256:2a625e4afab51b49edb0e5f4ff37d8afbb20ec644ed1e68641358a6305557de3"
]

I get

> skopeo inspect containers-storage:localhost/test:latest | jq .Layers
[
  "sha256:314640f419c581ddcac8f3618af39342a4571d5dc7a4e1f5b64d60f37e630b49",
  "sha256:244b70026015a275cd28c61f92ed398574006efe3bae6eeffddd904f168cfae5"
]

Describe the results you expected

With

> skopeo inspect docker://registry.access.redhat.com/ubi9:latest| jq .Layers
[
  "sha256:2a625e4afab51b49edb0e5f4ff37d8afbb20ec644ed1e68641358a6305557de3"
]

I get

skopeo inspect containers-storage:localhost/test:latest | jq .Layers [ "sha256:314640f419c581ddcac8f3618af39342a4571d5dc7a4e1f5b64d60f37e630b49", "sha256:244b70026015a275cd28c61f92ed398574006efe3bae6eeffddd904f168cfae5" ]

podman info output

> podman info
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.6-3.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: '
  cpuUtilization:
    idlePercent: 63.66
    systemPercent: 7.23
    userPercent: 29.1
  cpus: 4
  distribution:
    distribution: fedora
    variant: workstation
    version: "37"
  eventLogger: journald
  hostname: x220
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.1.12-200.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1271635968
  memTotal: 16653758464
  networkBackend: cni
  ociRuntime:
    name: crun
    package: crun-1.8.1-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.1
      commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8326721536
  swapTotal: 8326737920
  uptime: 84h 11m 39.00s (Approximately 3.50 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/tob/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/tob/.local/share/containers/storage
  graphRootAllocated: 122415972352
  graphRootUsed: 116710948864
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 32
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/tob/.local/share/containers/storage/volumes
version:
  APIVersion: 4.4.2
  Built: 1677669779
  BuiltTime: Wed Mar  1 12:22:59 2023
  GitCommit: ""
  GoVersion: go1.19.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.2



### Podman in a container

No

### Privileged Or Rootless

Rootless

### Upstream Latest Release

No

### Additional environment details

Additional environment details

### Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

flouthoc commented 1 year ago

I am not entirely sure but I think the layer digest changes when they are uncompressed, so parent layer is still there but the digest has changed. cc @mtrmac for info.

mtrmac commented 1 year ago

Yes, this is expected.

The layer digests in skopeo inspect report the specific representation of the layers. This can be confirmed e.g. by using skopeo inspect --raw --config, looking at rootfs.diffids, if the image format in question contains that field, or probably much more conveniently using Podman’s tools like podman image tree (which, OTOH can’t directly report about on-registry images).

(I would also mildly suggest that it’s not very clear to me why comparing the layer digests is useful to end users; e.g. the digests couldn’t possibly match when build --squash is used. What are you ultimately trying to do?)

ibotty commented 1 year ago

How am I supposed to check whether the base image is up to date? All documentation I can find will compare the layers.

BTW: It also kills deduplication, when I already have the base image and try to pull the image it pulls every image layer.

mtrmac commented 1 year ago

BTW: It also kills deduplication, when I already have the base image and try to pull the image it pulls every image layer.

That might be unexpected — but if the two are differently compressed, each computer must pull the two versions at least once to build a metadata database to know that they match. (And that can be determined using skopeo inspect of the two on-registry images.)

ibotty commented 1 year ago

I don't quiet understand. If it can be determined using skopeo inspect it should not be neccessary to pull them twice, because afaik skopeo inspect only downloads metadata.

Is it a given, that it's happening because of different compression? But coming back to the question, how can I determine if the base layer has been updated?

mtrmac commented 1 year ago

How am I supposed to check whether the base image is up to date?

that really rather depends on what assumptions can be made. As mentioned above, --squash would break the layer digest checks. The org.opencontainers.image.base.* annotations are another way, if you trust the image producer, and the image is using the OCI format.

mtrmac commented 1 year ago

WRT depuplication, please provide a complete reproducer, with debug logs of pulling the two images in sequence (podman --log-level=debug pull…), and if the two images aren’t public, all layer information from full outputs of skopeo inspect -n and skopeo inspect --raw --config.

ibotty commented 1 year ago

That squashing will kill the information is clear. Is there some documentation regarding the org.opencontainers.image.base.* annotations?

ibotty commented 1 year ago

WRT depuplication, please provide a complete reproducer, with debug logs of pulling the two images in sequence (podman --log-level=debug pull…), and if the two images aren’t public, all layer information from full outputs of skopeo inspect -n and skopeo inspect --raw --config.

I did (using registry.access.redhat.com/ubi9-minimal and quay.io/ibotty/s2i-deno) and if I read the log correctly, you are right, it does not seem to download the layer twice.

I am still not sure how to check if the base image is up-to-date. It seems podman and buildah do not set the org.opencontainers.image.base annotations by default. Can it be instructed to do so?

ibotty commented 1 year ago

Is is a common way to check for base image layers by comparing the layer images, see e.g. the github action https://github.com/twiddler/is-my-docker-parent-image-out-of-date, but there are others that use the same approach.

Is this approach fundamentally flawed?

I'd like to use the opencontainers annotations, but I could not find a tool that would show them. Can this maybe be integrated in skopeo?

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@mtrmac @vrothberg Could you answer @ibotty questions?

vrothberg commented 1 year ago

I'd like to use the opencontainers annotations, but I could not find a tool that would show them. Can this maybe be integrated in skopeo?

Both Podman and Buildah set the annotation, see below example:

~ $ podman build -f /tmp/Dockerfile
STEP 1/2: FROM ubi9
Resolved "ubi9" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull registry.access.redhat.com/ubi9:latest...
Getting image source signatures
Checking if image destination supports signatures
Copying blob 72d37ae8760a done  
Copying config 8e9c11168e done  
Writing manifest to image destination
Storing signatures
STEP 2/2: RUN echo "Issue #17722" > /issue
COMMIT
--> e662dd1af7ee
e662dd1af7eec0f131749b2670dd45bad763163541f98c1da2471605a6455490
~ $ podman image inspect e662|grep "org.opencontainers.image.base"
               "org.opencontainers.image.base.digest": "sha256:0bda22a725bb25c05b201ec250967a53fc4f3c6a7c82fcb2d8f6d3da3bed63dd",
               "org.opencontainers.image.base.name": "registry.access.redhat.com/ubi9:latest"

You can inspect the annotations via skopeo inspect --raw as shown below:

~ $ skopeo inspect --raw containers-storage:e66 | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:e662dd1af7eec0f131749b2670dd45bad763163541f98c1da2471605a6455490",
    "size": 6679
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:11939111cd6623e79e2b583306543b67986f0fb22b0c051ac16f563a2f7672c2",
      "size": 219357696
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar",
      "digest": "sha256:26a5a55a3a48d7b565f50013d012eefd611cb42bd0017fc9853135b3e92c99a9",
      "size": 4096
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.digest": "sha256:0bda22a725bb25c05b201ec250967a53fc4f3c6a7c82fcb2d8f6d3da3bed63dd",
    "org.opencontainers.image.base.name": "registry.access.redhat.com/ubi9:latest"
  }
}

With that information, you can use the base.name and compare the base.digest to the current digest on the registry. If they, differ then the image on the registry is newer. Be careful when comparing digests to always compare the digests of the specific image. For instance, to compare the digests of the linux/amd64 images.

containers / podman