search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.7k stars 2.41k forks source link

podman 3.2.0-rc1 fails on "podman build" if "--pull" is given #10307

Closed Spindel closed 3 years ago

Spindel commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman build --pull always fails as it attempts to pull intermediate container ID's while building.

Steps to reproduce the issue:

1 .podman build --file=Dockerfile --pull --tag=test/humbug:demo .

Describe the results you received:

STEP 1: FROM scratch
STEP 2: ARG URL=unknown
--> Using cache 4c2d990e1fc6149195126eb9278d8d924800d4ad1744a736978185f3bc163ec0
--> 4c2d990e1fc
ERRO[0000] error deleting build container "7786c68f63595e2b7770a179bad5822aac591169b1d883740c79b5e0ac46c621": identifier is not a container 
Error: identifier is not a container: error preparing container for next step: error creating build container: pull policy is always but image has been referred to by ID (4c2d990e1fc6149195126eb9278d8d924800d4ad1744a736978185f3bc163ec0)

Describe the results you expected:

A built container

Additional information you deem important (e.g. issue happens only occasionally):

Fedora 34 with stock podman.

Output of podman version:

Version:      3.2.0-rc1
API Version:  3.2.0-rc1
Go Version:   go1.16.3
Built:        Wed May  5 23:03:46 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

(paste your output here)

Package info (e.g. output of rpm -q podman or apt list podman):

host:
  arch: amd64
  buildahVersion: 1.20.2-dev
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.27-2.fc34.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 16
  distribution:
    distribution: fedora
    version: "34"
  eventLogger: journald
  hostname: tide.local
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.11.18-300.fc34.x86_64
  linkmode: dynamic
  memFree: 15293026304
  memTotal: 33590902784
  ociRuntime:
    name: crun
    package: crun-0.19.1-2.fc34.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1
      commit: 1535fedf0b83fb898d449f9680000f729ba719f5
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.9-1.fc34.x86_64
    version: |-
      slirp4netns version 1.1.8+dev
      commit: 6dc0186e020232ae1a6fcc1f7afbc3ea02fd3876
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 1h 0m 1.12s (Approximately 0.04 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /home/spider/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 0
    stopped: 5
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.fc34.x86_64
      Version: |-
        fusermount3 version: 3.10.2
        fuse-overlayfs: version 1.5
        FUSE library version 3.10.2
        using FUSE kernel interface version 7.31
  graphRoot: /home/spider/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 314
  runRoot: /run/user/1000
  volumePath: /home/spider/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.0-rc1
  Built: 1620248626
  BuiltTime: Wed May  5 23:03:46 2021
  GitCommit: ""
  GoVersion: go1.16.3
  OsArch: linux/amd64
  Version: 3.2.0-rc1

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Luap99 commented 3 years ago

@vrothberg PTAL

vrothberg commented 3 years ago

Thanks for reaching out!

The error seems legit to me. --pull with from scratch conflict IHMO. @nalind @rhatdan WDYT?

Spindel commented 3 years ago

it also happens without "FROM scratch" scratch was just my minimal example.

Spindel commented 3 years ago

Thanks for reaching out!

The error seems legit to me. --pull with from scratch conflict IHMO. @nalind @rhatdan WDYT?

Ex from our public CI: https://gitlab.com/ModioAB/containers/-/jobs/1251708428

The Dockerfile is this one: https://gitlab.com/ModioAB/containers/-/blob/master/mongodb/Dockerfile

Which is a fairly simple example.

vrothberg commented 3 years ago

Thanks, I am looking at it now.

vrothberg commented 3 years ago

Okay, that's Podman-side bug. buildah bud works just fine.

rhatdan commented 3 years ago

Could we have a Dockerfile for this. If I run 

$ cat Dockerfile 
FROM scratch
ARG URL=unknown
$ podman build --no-cache --file=Dockerfile --pull --tag=test/humbug:demo .
STEP 1: FROM scratch
STEP 2: ARG URL=unknown
STEP 3: COMMIT test/humbug:demo
--> 16c63e0ff5c
Successfully tagged localhost/test/humbug:demo
16c63e0ff5c24338ca5aa651991b26632c59b96b231e1142a39a5d6054b00dba

It works for me?

vrothberg commented 3 years ago

The Dockerfile is this one: https://gitlab.com/ModioAB/containers/-/blob/master/mongodb/Dockerfile

@rhatdan, try this ^ one.

Spindel commented 3 years ago 
FROM  docker.io/centos/mongodb-36-centos7:latest
ARG DATE=unknown
LABEL "se.modio.ci.date"=$DATE
ENV LANG C.UTF-8
ENV LC_CTYPE C.utf8
$ podman --version
podman version 3.2.0-rc1
$ podman build --file=Dockerfile --pull --tag=test/humbug:demo

STEP 1: FROM docker.io/centos/mongodb-36-centos7:latest
Trying to pull docker.io/centos/mongodb-36-centos7:latest...
Getting image source signatures
Copying blob 21fea6b5d8d6 skipped: already exists  
Copying blob 23aa851937ef skipped: already exists  
Copying blob 1e910e5e32e0 skipped: already exists  
Copying blob e5783daef160 skipped: already exists  
Copying blob b2cc5146c9c7 skipped: already exists  
Copying blob 8b952114afd6 skipped: already exists  
Copying blob 0e412082be0c skipped: already exists  
Copying blob 93489c8b2192 [--------------------------------------] 0.0b / 0.0b
Copying config 90cc59a937 done  
Writing manifest to image destination
Storing signatures
STEP 2: ARG DATE=unknown
--> Using cache 2f1bca23b7a087b96bd5adc11641df589c47913ca2d96dde572de323a7b09ec3
--> 2f1bca23b7a
ERRO[0002] error deleting build container "856a492739ab4e9c26e6aa4fa79ce838b855bcac3319cecfafcb4b2a6c5c0199": identifier is not a container 
Error: identifier is not a container: error preparing container for next step: error creating build container: pull policy is always but image has been referred to by ID (2f1bca23b7a087b96bd5adc11641df589c47913ca2d96dde572de323a7b09ec3)
TomSweeneyRedHat commented 3 years ago

This last example seems to work for me with upstream. I'm not sure what's going on here....

# podman version
API Version:  3.2.0-dev
Go Version:   go1.15.8
Git Commit:   6ff56ab50a10e2350013c4ff1bf4b8d5d7b5aa87
Built:        Tue May 11 14:00:46 2021
OS/Arch:      linux/amd64

# podman build --file=Tomfile --pull --tag=test/humbug:demo
STEP 1: FROM docker.io/centos/mongodb-36-centos7:latest
Getting image source signatures
Copying blob 1e910e5e32e0 done  
Copying blob 8b952114afd6 done  
Copying blob 23aa851937ef done  
Copying blob b2cc5146c9c7 done  
Copying blob e5783daef160 done  
Copying blob 21fea6b5d8d6 done  
Copying blob 0e412082be0c done  
Copying blob 93489c8b2192 done  
Copying config 90cc59a937 done  
Writing manifest to image destination
Storing signatures
STEP 2: ARG DATE=unknown
--> c2e2d40d623
STEP 3: LABEL "se.modio.ci.date"=$DATE
--> d25b7953cde
STEP 4: ENV LANG C.UTF-8
--> 4beeeabec18
STEP 5: ENV LC_CTYPE C.utf8
STEP 6: COMMIT test/humbug:demo
--> 952d97145eb
952d97145ebab2f1ac6bbdb343cef0119258a0b5537b5972af8726696f8f6ed7
Spindel commented 3 years ago

I haven't tested with upstream, only with the current fedora (33 and 34) package, which is:

Version:      3.2.0-rc1
API Version:  3.2.0-rc1
Go Version:   go1.16.3
Built:        Wed May  5 23:03:46 2021
OS/Arch:      linux/amd64
vrothberg commented 3 years ago

I reproduced it successfully and already started looking into it.

vrothberg commented 3 years ago

I opened https://github.com/containers/buildah/pull/3222 to fix the issue. The extended test will also be executed in Podman once vendored.