Manifest generation on building multi-platform from scratch

ReDemoNBR commented 1 year ago

Issue Description

When building multiarch images from scratch, the manifest list generated only shows 1 of the images. The error is not reproduced when using an image like alpine:latest

Note that podman v4.5.1 used here includes buildah=1.30.0. Using buildah in the very same version 1.30.0 does not reproduce the issue.

Using on ArchLinux amd64

Steps to reproduce the issue

Build a multi-arch image from scratch: podman build --platform linux/amd64,linux/arm64/v8 --manifest foo:bar .
Inspect the manifest list: podman manifest inspect foo:bar
Assert that one of the platform-specific images is not included in the manifest list

Describe the results you received

Using scratch as base image

FROM scratch
COPY hello.txt /hello.txt

$ podman build --platform linux/amd64,linux/arm64/v8 --manifest mytest:scratch
$ podman manifest inspect mytest:scratch
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 495,
            "digest": "...",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    ]
}

The manifest list is missing the linux/arm64/v8 image

Describe the results you expected

Using alpine as base image

If the base image is changed, for example to alpine:latest like so:

FROM docker.io/alpine:latest
COPY hello.txt /hello.txt

It generates the manifest correctly

$ podman build --platform linux/amd64,linux/arm64/v8 --manifest mytest:alpine
$ podman manifest inspect mytest:alpine
{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
    "manifests": [
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 750,
            "digest": "...",
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        },
        {
            "mediaType": "application/vnd.oci.image.manifest.v1+json",
            "size": 750,
            "digest": "...",
            "platform": {
                "architecture": "arm64",
                "os": "linux",
                "variant": "v8"
            }
        }
    ]
}

The manifest list is not missing any of the images

podman info output

host:
  arch: amd64
  buildahVersion: 1.30.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.7-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: f633919178f6c8ee4fb41b848a056ec33f8d707d'
  cpuUtilization:
    idlePercent: 98.88
    systemPercent: 0.29
    userPercent: 0.83
  cpus: 32
  databaseBackend: boltdb
  distribution:
    distribution: endeavouros
    version: unknown
  eventLogger: journald
  hostname: redmachine
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 55537
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 55537
  kernel: 6.3.4-arch1-1
  linkmode: dynamic
  logDriver: journald
  memFree: 1142894592
  memTotal: 67341844480
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.8.5-1
    path: /usr/bin/crun
    version: |-
      crun version 1.8.5
      commit: b6f80f766c9a89eb7b1440c0a70ab287434b17ed
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.0-1
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 0
  swapTotal: 0
  uptime: 2h 44m 54.00s (Approximately 0.08 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/rdn/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/rdn/.local/share/containers/storage
  graphRootAllocated: 2013517959168
  graphRootUsed: 243525132288
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 108
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/rdn/.local/share/containers/storage/volumes
version:
  APIVersion: 4.5.1
  Built: 1685139594
  BuiltTime: Fri May 26 19:19:54 2023
  GitCommit: 9eef30051c83f62816a1772a743e5f1271b196d7-dirty
  GoVersion: go1.20.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.5.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

Luap99 commented 1 year ago

@flouthoc PTAL

flouthoc commented 1 year ago

@ReDemoNBR Ah this happens because buildah never re-adds same image again since image is built from scratch and cache is used, if you use --no-cache with the command it should work.

Anyways this must be fixed as buildah end. I'll take a look.

ReDemoNBR commented 1 year ago

@flouthoc Thanks, this makes sense.

I just retried my initial steps without --no-cache and sometimes buildah does fail and sometimes it works, differently than my initial report says. It is plain inconsistent. Sometimes re-running the build without destroying the previous build, it even ends up with some weird stuff like linux/amd/v8 lol when the second attempt adds the linux/amd but the previously built was linux/arm64/v8.

--no-cache indeed fixes the outputs, but this does have some effect on intermediary build stages that are shared. But surely fixed the issue.

As this is a problem upstream, should I reopen and link this issue on buildah repository?

flouthoc commented 1 year ago

I think issue is we can't add same digest twice in the manifest list.

mtrmac commented 1 year ago

If the build cache does not match architectures (?), it probably should start doing so
in principle, I think a manifest list is allowed to contain the same digest twice (e.g., hypothetically, it could have the same digest in an arm/v7 and an arm/v8 entry — it would be redundant but I don’t see why it would be invalid). If our internal representation makes that impossible, we might consider changing it.

github-actions[bot] commented 1 year ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 1 year ago

@flouthoc Any update on this?

containers / podman