`podman build` ignores existing tags, doesn't match Docker behaviour when `--platform` is used

srcshelton commented 8 months ago

Issue Description

When running podman build on a Containerfile, even with pull=never a tagged image is replaced with its upstream version.

Regardless of the --pull directive, Docker uses the existing tagged image - which is the behaviour I'd expect from podman.

N.B. This only happens when a --platform argument is supplied - if not, the behaviour is as expected.

Steps to reproduce the issue

# podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc
# podman pull docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25
Trying to pull docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25...
Getting image source signatures
Copying blob ab438529e560 skipped: already exists  
Copying config de2f56d15f done   | 
Writing manifest to image destination
de2f56d15f7149eece372c60213e087bdb7183ac621e2cad72ed843d6903ef0b
# podman tag docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25 docker.io/gentoo/stage3:amd64-nomultilib-openrc
# podman image ls | grep openrc
docker.io/gentoo/stage3                             amd64-nomultilib-openrc  de2f56d15f71  3 weeks ago    1.3 GB
# cat > Containerfile <<'EOF'
ARG gentoo_stage3_image="docker.io/gentoo/stage3"
ARG gentoo_stage3_tag="latest"

FROM "${gentoo_stage3_image}:${gentoo_stage3_tag}" AS stage3
EOF
# podman image build --platform linux/amd64 --build-arg ARCH=amd64 --build-arg env_name=localhost/gentoo-env --build-arg env_id=70e487ac92ed --build-arg gentoo_stage3_image=docker.io/gentoo/stage3 --build-arg gentoo_stage3_tag=amd64-nomultilib-openrc --build-arg gentoo_stage3_id=de2f56d15f71 --compress --file Containerfile --network none --no-cache --pull=false --tag localhost/gentoo-stage3:latest --build-arg stage3_flags_file=/usr/libexec/stage3_flags.sh --loglevel 3 .
STEP 1/1: FROM docker.io/gentoo/stage3:amd64-nomultilib-openrc AS stage3
Trying to pull docker.io/gentoo/stage3:amd64-nomultilib-openrc...
Getting image source signatures
Copying blob 6185d679c3f6 skipped: already exists  
Copying config ee43876c4a done   | 
Writing manifest to image destination
COMMIT localhost/gentoo-stage3:latest
--> ee43876c4a27
[Warning] one or more build args were not consumed: [ARCH env_id env_name gentoo_stage3_id stage3_flags_file]
Successfully tagged localhost/gentoo-stage3:latest
Successfully tagged docker.io/gentoo/stage3:amd64-nomultilib-openrc
ee43876c4a27f295f06bba26ef842156ff2cdd3f645e73cfe5c12126e49798e5
# podman image ls | grep openrc
docker.io/gentoo/stage3                             amd64-nomultilib-openrc  ee43876c4a27  2 days ago      1.29 GB

N.B. de2f56d15f71 has become ee43876c4a27 - it shouldn't have changed!

Again with more minimal arguments to podman build:

# podman image untag docker.io/gentoo/stage3:amd64-nomultilib-openrc
# podman pull docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25
Trying to pull docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25...
Getting image source signatures
Copying blob ab438529e560 skipped: already exists  
Copying config de2f56d15f done   | 
Writing manifest to image destination
de2f56d15f7149eece372c60213e087bdb7183ac621e2cad72ed843d6903ef0b
# podman tag docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25 docker.io/gentoo/stage3:amd64-nomultilib-openrc
# podman image ls | grep openrc
docker.io/gentoo/stage3                             amd64-nomultilib-openrc  de2f56d15f71  3 weeks ago    1.3 GB
# podman image build --platform linux/amd64 --build-arg gentoo_stage3_image=docker.io/gentoo/stage3 --build-arg gentoo_stage3_tag=amd64-nomultilib-openrc --file Containerfile --pull=false --tag localhost/gentoo-stage3:latest .
STEP 1/1: FROM docker.io/gentoo/stage3:amd64-nomultilib-openrc AS stage3
Trying to pull docker.io/gentoo/stage3:amd64-nomultilib-openrc...
Getting image source signatures
Copying blob 6185d679c3f6 skipped: already exists  
Copying config ee43876c4a done   | 
Writing manifest to image destination
COMMIT localhost/gentoo-stage3:latest
--> ee43876c4a27
Successfully tagged localhost/gentoo-stage3:latest
Successfully tagged docker.io/gentoo/stage3:amd64-nomultilib-openrc
ee43876c4a27f295f06bba26ef842156ff2cdd3f645e73cfe5c12126e49798e5
5c12126e49798e5
# podman image ls | grep openrc
docker.io/gentoo/stage3                             amd64-nomultilib-openrc  ee43876c4a27  2 days ago      1.29 GB

… vs:

…
podman image build --build-arg gentoo_stage3_tag=amd64-nomultilib-openrc --file Containerfile --pull=false --tag localhost/gentoo-stage3:latest .
STEP 1/1: FROM docker.io/gentoo/stage3:amd64-nomultilib-openrc AS stage3
COMMIT localhost/gentoo-stage3:latest
--> de2f56d15f71
Successfully tagged localhost/gentoo-stage3:latest
Successfully tagged docker.io/gentoo/stage3:amd64-nomultilib-openrc
Successfully tagged docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25
de2f56d15f7149eece372c60213e087bdb7183ac621e2cad72ed843d6903ef0b
# podman image ls | grep openrc
docker.io/gentoo/stage3                             amd64-nomultilib-openrc  de2f56d15f71  3 weeks ago     1.3 GB

Describe the results you received

With --platform, the tagged image is always replaced with the upstream version even if --pull=never is supplied.

Describe the results you expected

Without --platform, the behaviour is as anticipated and matches Docker behaviour (which doesn't vary depending on --platform).

podman info output

# podman info
host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: app-containers/conmon-2.1.10
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.10, commit: 3a9715d28cb4cf0e671dfbc4211d4458534db189'
  cpuUtilization:
    idlePercent: 97.46
    systemPercent: 0.88
    userPercent: 1.66
  cpus: 8
  databaseBackend: sqlite
  distribution:
    distribution: gentoo
    version: "2.14"
  eventLogger: file
  freeLocks: 1995
  hostname: dellr330
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 6.7.3-gentoo
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 9738526720
  memTotal: 67328946176
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: app-containers/aardvark-dns-1.10.0
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: app-containers/netavark-1.10.3
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: app-containers/crun-1.14
    path: /usr/bin/crun
    version: |-
      crun version 1.14
      commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
      rundir: /var/run/crun
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: unix:///var/run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 42949652480
  swapTotal: 42949652480
  uptime: 80h 9m 6.00s (Approximately 3.33 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  localhost:5000:
    Blocked: false
    Insecure: true
    Location: localhost:5000
    MirrorByDigestOnly: false
    Mirrors: null
    Prefix: localhost:5000
    PullFromMirror: ""
  search:
  - docker.io
  - docker.pkg.github.com
  - quay.io
  - public.ecr.aws
  - registry.fedoraproject.org
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 27
    paused: 0
    running: 24
    stopped: 3
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /mnt/podman-storage
  graphRootAllocated: 2000397795328
  graphRootUsed: 57536409600
  graphStatus:
    Build Version: Btrfs v6.6.3
    Library Version: "102"
  imageCopyTmpDir: /var/tmp/.private/root
  imageStore:
    number: 245
  runRoot: /var/run/podman
  transientStore: false
  volumePath: /space/podman/volumes
version:
  APIVersion: 4.9.3
  Built: 1707865915
  BuiltTime: Tue Feb 13 23:11:55 2024
  GitCommit: ""
  GoVersion: go1.21.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.3

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

Yes

Additional environment details

Bare-metal host

Additional information

No response

srcshelton commented 8 months ago

N.B. This can be worked-around with:

FROM docker.io/gentoo/stage3@sha256:cd31a571b4ac0af92cee68611210df035b327caa1ae7f5cb242f893304b21f25 AS stage3

… but surely the options above should also work?

rhatdan commented 8 months ago

Change --pull=false to --pull=never

srcshelton commented 8 months ago

Change --pull=false to --pull=never

It looks as if there's a fix PR posted, but I'll quickly note the following:

# podman build --help | grep pull
      --pull string[="true"]                         Pull image policy ("always/true"|"missing"|"never/false"|"newer") (default "missing")

The other thing, and I'm not sure how much this project is concerned by this, is that the default Docker behaviour seems to be not to overwrite an existing tag if it exists.

N.B. My testing made it look as if this is only an issue when --platform is used, which doesn't appear to relate to the PR raised… have all occurrences related to this issue been addressed by https://github.com/containers/buildah/pull/5354, or are there more to come?

containers / podman