Weird cache behavior with `podman build --omit-history`

Issue Description

Cache interaction is broken when building images via podman build --omit-history.

New layers don't end up in the cache, and existing layers are reused including their history.

Steps to reproduce the issue

$ printf 'FROM docker.io/library/alpine:3.18\nRUN date' | podman build --omit-history -
STEP 1/2: FROM docker.io/library/alpine:3.18
Trying to pull docker.io/library/alpine:3.18...
Getting image source signatures
Copying blob 96526aa774ef skipped: already exists
Copying config 8ca4688f4f done   |
Writing manifest to image destination
STEP 2/2: RUN date
Tue Nov 21 15:32:52 UTC 2023
COMMIT
--> f7e920a5e917
f7e920a5e9171144da5797b109a2c136cfcb9c3d25570a2e15238ddccd18be18

$ printf 'FROM docker.io/library/alpine:3.18\nRUN date' | podman build --omit-history -
STEP 1/2: FROM docker.io/library/alpine:3.18
STEP 2/2: RUN date
Tue Nov 21 15:32:54 UTC 2023
COMMIT
--> 285f5f14252c
285f5f14252cee7d3eb88f75957adac208528a5471d17b40705ee4b1d2b010fa

$ printf 'FROM docker.io/library/alpine:3.18\nRUN date' | podman build --omit-history -
STEP 1/2: FROM docker.io/library/alpine:3.18
STEP 2/2: RUN date
Tue Nov 21 15:32:57 UTC 2023
COMMIT
--> ee3af5ae117e
ee3af5ae117efefe9b334b4e2cbad23ffb92879b93e8a794c6e3bfb54a5977af

$ podman inspect ee3af5ae117efefe9b334b4e2cbad23ffb92879b93e8a794c6e3bfb54a5977af | jq '.[0].History'
null

$ printf 'FROM docker.io/library/alpine:3.18\nRUN date' | podman build  -
STEP 1/2: FROM docker.io/library/alpine:3.18
STEP 2/2: RUN date
Tue Nov 21 15:33:03 UTC 2023
COMMIT
--> 4b4fcc775992
4b4fcc775992121e70781051b7b294ee8b175bd24672b6c632131f03f81c7f8d

$ printf 'FROM docker.io/library/alpine:3.18\nRUN date' | podman build  -
STEP 1/2: FROM docker.io/library/alpine:3.18
STEP 2/2: RUN date
--> Using cache 4b4fcc775992121e70781051b7b294ee8b175bd24672b6c632131f03f81c7f8d
--> 4b4fcc775992
4b4fcc775992121e70781051b7b294ee8b175bd24672b6c632131f03f81c7f8d

$ printf 'FROM docker.io/library/alpine:3.18\nRUN date' | podman build --omit-history -
STEP 1/2: FROM docker.io/library/alpine:3.18
STEP 2/2: RUN date
--> Using cache 4b4fcc775992121e70781051b7b294ee8b175bd24672b6c632131f03f81c7f8d
--> 4b4fcc775992
4b4fcc775992121e70781051b7b294ee8b175bd24672b6c632131f03f81c7f8d

$ podman inspect 4b4fcc775992121e70781051b7b294ee8b175bd24672b6c632131f03f81c7f8d | jq '.[0].History'
[
  {
    "created": "2023-09-28T21:19:27.686110063Z",
    "created_by": "/bin/sh -c #(nop) ADD file:756183bba9c7f4593c2b216e98e4208b9163c4c962ea0837ef88bd917609d001 in / "
  },
  {
    "created": "2023-09-28T21:19:27.801479409Z",
    "created_by": "/bin/sh -c #(nop)  CMD [\"/bin/sh\"]",
    "empty_layer": true
  },
  {
    "created": "2023-11-21T15:33:04.000783167Z",
    "created_by": "/bin/sh -c date",
    "comment": "FROM docker.io/library/alpine:3.18"
  }
]

Describe the results you received

When starting a fresh image build using --omit-history, everything works, but the layer cache won't get populated. I verified this by running the same command multiple times. Doing the same thing without the --omit-history flag yields a cache hit on the second invocation. Funnily enough, a subsequent invocation including the --omit-history flag now has cache hits as well, but this time, the images actually include the history.

Describe the results you expected

Proper caching for builds, no matter if they're using --omit-history or not.
No history for images built with --omit-history, no matter what.

podman info output

host:
  arch: amd64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /nix/store/8xbnlbw5h8xfbg1k5rblj47vxy6h7jq4-conmon-2.1.8/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 91.55
    systemPercent: 3.11
    userPercent: 5.34
  cpus: 16
  databaseBackend: boltdb
  distribution:
    codename: stoat
    distribution: nixos
    version: "23.05"
  eventLogger: journald
  freeLocks: 2048
  hostname: miratom
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.5.11
  linkmode: dynamic
  logDriver: journald
  memFree: 1987600384
  memTotal: 15958720512
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: Unknown
      path: /nix/store/j4hvddv6ps6apdfbh9604nrmk77jnivj-podman-4.7.2/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: Unknown
    path: /nix/store/j4hvddv6ps6apdfbh9604nrmk77jnivj-podman-4.7.2/libexec/podman/netavark
    version: netavark 1.7.0
  ociRuntime:
    name: crun
    package: Unknown
    path: /nix/store/85p6grnbbh2imk49b5ckz81956ywax05-crun-1.11.1/bin/crun
    version: |-
      crun version 1.11.1
      commit: 1.11.1
      rundir: /run/user/2000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/2000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: ""
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /nix/store/j4hvddv6ps6apdfbh9604nrmk77jnivj-podman-4.7.2/libexec/podman/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 30218821632
  swapTotal: 34359734272
  uptime: 100h 18m 36.00s (Approximately 4.17 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/twieczorek/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/twieczorek/.local/share/containers/storage
  graphRootAllocated: 338417418240
  graphRootUsed: 185008136192
  graphStatus:
    Build Version: Btrfs v6.6.2
    Library Version: "102"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 5
  runRoot: /run/user/2000/containers
  transientStore: false
  volumePath: /home/twieczorek/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.2
  Built: 315532800
  BuiltTime: Tue Jan  1 01:00:00 1980
  GitCommit: ""
  GoVersion: go1.21.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.7.2

Podman in a container

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

NixOS 23.05. System on commit 9fb1225, using the podman binary from current master (9075a0d).

Additional information

Using caches with history for builds using --omit-history seems reasonable, but the history would need to be stripped afterwards, of course.

containers / podman