search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.51k stars 2.39k forks source link

PSA: Unable to run pod after upgrade from podman:amd64 (2.0.5~2, 2.0.6~2) on Ubuntu 20.04 #7663

Closed averyfreeman closed 3 years ago

averyfreeman commented 4 years ago

BUG REPORT or FEATURE REQUEST?

/kind bug

Description

Steps to reproduce:

  1. Upgrade podman amd64 from 2.0.5\~2 to 2.0.6\~2 via apt/dpkg

  2. Reboot machine (not sure if relevant, but was step I took)

  3. run pod $ podman pod start <podname>

Received result:

error starting container /proc/16142/ns/pid OCI runtime command not found

Expected result:

podman starting pod

Other info:

I'm submitting this bug report just so you're aware of the issue I had, even though I managed to fix it (somewhat clunkily).

The pod start error happened consistently (there were no alternative outcomes). Always looking for same pid.

Was attempting to start pod as user (not root).

I'm using overlayfs and crun:

crun/unknown,now 0.14.1~3 amd64
fuse-overlayfs/unknown,now 1.1.2~1 amd64

I tried deleting /run/user/$UID/libpod/pause.pid, same outcome

I ended up deleting ~/.local/share/containers/storage/ and ~/.config/containers/storage/ with simple rm -rf, and re-importing my pod tar.

Output of podman version:

podman version 2.0.6

Podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.1
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 1
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: afw0
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 10002
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 10001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.4.0-47-generic
  linkmode: dynamic
  memFree: 768364544
  memTotal: 2084274176
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 3e46dd849fdf6bfa68127786e073318184641f05
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/10001/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 2143285248
  swapTotal: 2147479552
  uptime: 3h 46m 0.51s (Approximately 0.12 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/localuser/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
    overlay.mountopt: nodev
  graphRoot: /home/localuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /home/localuser/.local/share/containers/storage
  volumePath: /home/localuser/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.6

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.0.6~2 amd64 [installed]
podman/unknown 2.0.6~2 arm64
podman/unknown 2.0.6~2 armhf
podman/unknown 2.0.6~2 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes, both

Additional environment details (AWS, VirtualBox, physical, etc.): hyper-expert VPS (appears to be KVM 440FX VM w/ virtio net + disk):

lshw output:

afw0                        
    description: Computer
    product: KVM
    vendor: Red Hat
    version: RHEL 7.6.0 PC (i440FX + PIIX, 1996)
    width: 64 bits
    capabilities: smbios-2.8 dmi-2.8 vsyscall32
    configuration: boot=normal family=Red Hat Enterprise Linux uuid=C4732E29-A873-F44A-9D16-D4FA954913A9
  *-core
       description: Motherboard
       physical id: 0
     *-firmware
          description: BIOS
          vendor: SeaBIOS
          physical id: 0
          version: 1.11.0-2.el7
          date: 04/01/2014
          size: 96KiB
     *-cpu
          description: CPU
          product: Intel(R) Xeon(R) CPU E5-2637 v2 @ 3.50GHz
          vendor: Intel Corp.
          physical id: 400
          bus info: cpu@0
          version: RHEL 7.6.0 PC (i440FX + PIIX, 1996)
          slot: CPU 0
          size: 2GHz
          capacity: 2GHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp x86-64 constant_tsc arch_perfmon rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust smep erms xsaveopt arat md_clear
          configuration: cores=1 enabledcores=1 threads=1
     *-memory
          description: System Memory
          physical id: 1000
          size: 2GiB
          capacity: 2GiB
          capabilities: ecc
          configuration: errordetection=multi-bit-ecc
        *-bank
             description: DIMM RAM
             vendor: Red Hat
             physical id: 0
             slot: DIMM 0
             size: 2GiB
     *-pci
          description: Host bridge
          product: 440FX - 82441FX PMC [Natoma]
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 02
          width: 32 bits
          clock: 33MHz
        *-isa
             description: ISA bridge
             product: 82371SB PIIX3 ISA [Natoma/Triton II]
             vendor: Intel Corporation
             physical id: 1
             bus info: pci@0000:00:01.0
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: isa
             configuration: latency=0
        *-ide
             description: IDE interface
             product: 82371SB PIIX3 IDE [Natoma/Triton II]
             vendor: Intel Corporation
             physical id: 1.1
             bus info: pci@0000:00:01.1
             logical name: scsi1
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: ide isa_compat_mode bus_master emulated
             configuration: driver=ata_piix latency=0
             resources: irq:0 ioport:1f0(size=8) ioport:3f6 ioport:170(size=8) ioport:376 ioport:c0a0(size=16)
           *-cdrom
                description: DVD reader
                product: QEMU DVD-ROM
                vendor: QEMU
                physical id: 0.0.0
                bus info: scsi@1:0.0.0
                logical name: /dev/cdrom
                logical name: /dev/dvd
                logical name: /dev/sr0
                version: 2.5+
                capabilities: removable audio dvd
                configuration: ansiversion=5 status=nodisc
        *-usb
             description: USB controller
             product: 82371SB PIIX3 USB [Natoma/Triton II]
             vendor: Intel Corporation
             physical id: 1.2
             bus info: pci@0000:00:01.2
             version: 01
             width: 32 bits
             clock: 33MHz
             capabilities: uhci bus_master
             configuration: driver=uhci_hcd latency=0
             resources: irq:11 ioport:c040(size=32)
           *-usbhost
                product: UHCI Host Controller
                vendor: Linux 5.4.0-47-generic uhci_hcd
                physical id: 1
                bus info: usb@1
                logical name: usb1
                version: 5.04
                capabilities: usb-1.10
                configuration: driver=hub slots=2 speed=12Mbit/s
              *-usb
                   description: Human interface device
                   product: QEMU USB Tablet
                   vendor: QEMU
                   physical id: 1
                   bus info: usb@1:1
                   version: 0.00
                   serial: 42
                   capabilities: usb-2.00
                   configuration: driver=usbhid maxpower=100mA speed=12Mbit/s
        *-bridge
             description: Bridge
             product: 82371AB/EB/MB PIIX4 ACPI
             vendor: Intel Corporation
             physical id: 1.3
             bus info: pci@0000:00:01.3
             version: 03
             width: 32 bits
             clock: 33MHz
             capabilities: bridge
             configuration: driver=piix4_smbus latency=0
             resources: irq:9
        *-display
             description: VGA compatible controller
             product: GD 5446
             vendor: Cirrus Logic
             physical id: 2
             bus info: pci@0000:00:02.0
             version: 00
             width: 32 bits
             clock: 33MHz
             capabilities: vga_controller rom
             configuration: driver=cirrus latency=0
             resources: irq:0 memory:fc000000-fdffffff memory:febd0000-febd0fff memory:c0000-dffff
        *-network
             description: Ethernet controller
             product: Virtio network device
             vendor: Red Hat, Inc.
             physical id: 3
             bus info: pci@0000:00:03.0
             version: 00
             width: 64 bits
             clock: 33MHz
             capabilities: msix bus_master cap_list rom
             configuration: driver=virtio-pci latency=0
             resources: irq:10 ioport:c060(size=32) memory:febd1000-febd1fff memory:fe000000-fe003fff memory:feb80000-febbffff
           *-virtio0
                description: Ethernet interface
                physical id: 0
                bus info: virtio@0
                logical name: eth0
                serial: 00:16:3c:63:d3:d8
                capabilities: ethernet physical
                configuration: autonegotiation=off broadcast=yes driver=virtio_net driverversion=1.0.0 ip=208.87.97.100 link=yes multicast=yes
        *-scsi
             description: SCSI storage controller
             product: Virtio block device
             vendor: Red Hat, Inc.
             physical id: 4
             bus info: pci@0000:00:04.0
             version: 00
             width: 64 bits
             clock: 33MHz
             capabilities: scsi msix bus_master cap_list
             configuration: driver=virtio-pci latency=0
             resources: irq:11 ioport:c000(size=64) memory:febd2000-febd2fff memory:fe004000-fe007fff
           *-virtio1
                description: Virtual I/O device
                physical id: 0
                bus info: virtio@1
                logical name: /dev/vda
                size: 42GiB (45GB)
                capabilities: partitioned partitioned:dos
                configuration: driver=virtio_blk logicalsectorsize=512 sectorsize=512 signature=c8d5eddc
              *-volume:0
                   description: EXT4 volume
                   vendor: Linux
                   physical id: 1
                   bus info: virtio@1,1
                   logical name: /dev/vda1
                   logical name: /
                   logical name: /var/lib/containers/storage/overlay
                   version: 1.0
                   serial: 9de45e18-29e7-4330-b5ab-8a272f87aa36
                   size: 39GiB
                   capacity: 39GiB
                   capabilities: primary bootable journaled extended_attributes large_files huge_files dir_nlink recover 64bit extents ext4 ext2 initialized
                   configuration: created=2018-04-30 10:10:44 filesystem=ext4 lastmountpoint=/ modified=2020-09-16 19:58:25 mount.fstype=ext4 mount.options=rw,relatime,errors=remount-ro mounted=2020-09-16 19:58:27 state=mounted
              *-volume:1
                   description: Linux swap volume
                   physical id: 2
                   bus info: virtio@1,2
                   logical name: /dev/vda2
                   version: 1
                   serial: 7bd1040a-18f6-436e-9d26-c58865a7ca5f
                   size: 2GiB
                   capacity: 2GiB
                   capabilities: primary nofs swap initialized
                   configuration: filesystem=swap pagesize=4096
        *-generic
             description: Unclassified device
             product: Virtio memory balloon
             vendor: Red Hat, Inc.
             physical id: 5
             bus info: pci@0000:00:05.0
             version: 00
             width: 64 bits
             clock: 33MHz
             capabilities: bus_master cap_list
             configuration: driver=virtio-pci latency=0
             resources: irq:10 ioport:c080(size=32) memory:fe008000-fe00bfff
           *-virtio2 UNCLAIMED
                description: Virtual I/O device
                physical id: 0
                bus info: virtio@2
                configuration: driver=virtio_balloon
     *-pnp00:00
          product: PnP device PNP0b00
          physical id: 1
          capabilities: pnp
          configuration: driver=rtc_cmos
     *-pnp00:01
          product: PnP device PNP0303
          physical id: 2
          capabilities: pnp
          configuration: driver=i8042 kbd
     *-pnp00:02
          product: PnP device PNP0f13
          physical id: 3
          capabilities: pnp
          configuration: driver=i8042 aux
     *-pnp00:03
          product: PnP device PNP0700
          physical id: 4
          capabilities: pnp
  *-network:0
       description: Ethernet interface
       physical id: 1
       logical name: cni-podman0
       serial: ce:e6:25:31:bf:fd
       capabilities: ethernet physical
       configuration: broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=10.88.0.1 link=yes multicast=yes
  *-network:1
       description: Ethernet interface
       physical id: 2
       logical name: veth670ddc8f
       serial: 4e:f7:6e:94:a0:dc
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
lsm5 commented 4 years ago

@averyfreeman do you see this when you upgrade to 2.1.1 as well?

averyfreeman commented 4 years ago

@averyfreeman do you see this when you upgrade to 2.1.1 as well?

I haven't tried it yet. I will do some tests locally on an essentially identical setup and let you know what I find out. Thanks :)

github-actions[bot] commented 3 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 3 years ago

I figure this worked, so closing, reopen if I am mistaken.