podman: panic: runtime error: invalid memory address or nil pointer dereference

[petrkr]

Maintainer: Oskari Rauta Environment: ARMv7 - Turris Omnia - OpenWRT 21.02 (Turris OS 6.2.1) Description:

Seems same problem like this one https://github.com/containers/podman/issues/10535

Just wonder if update to latest 3.4.x podman version (https://github.com/containers/podman/releases?q=v3.4&expanded=true) would fix that. It is hard for me to set-up whole build environment to rebuild just one package to test it

Package makefile: https://git.openwrt.org/?p=feed/packages.git;a=blob;f=utils/podman/Makefile;h=e8323d4022e0b332a00a0ef4c7ba17e3fae0debf;hb=refs/heads/openwrt-21.02

i've tried overlay and btrfs (as there is Btrfs filesystem) storage engine, both are same.

root@turris:~# podman run -it --rm arm32v7/busybox
✔ docker.io/arm32v7/busybox:latest
Trying to pull docker.io/arm32v7/busybox:latest...
Getting image source signatures
Copying blob 46758452d3ee done  
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x200 pc=0x58e0a0]

goroutine 87 [running]:
reflect.mapiterinit(0x1bce3c0, 0x3f2a4c0, 0x416a4b0)
        runtime/map.go:1373 +0x2c
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
        github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:112
github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0x3e88738, 0x3eec388)
        github.com/json-iterator/go@v1.1.11/reflect_map.go:333 +0x34
github.com/json-iterator/go.(*structFieldEncoder).IsEmpty(0x3e88f90, 0x3eec300)
        github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:118 +0x50
github.com/json-iterator/go.(*structEncoder).Encode(0x3e89008, 0x3eec300, 0x416a4b0)
        github.com/json-iterator/go@v1.1.11/reflect_struct_encoder.go:148 +0x63c
github.com/json-iterator/go.(*OptionalEncoder).Encode(0x3ee8838, 0x3ee8158, 0x416a4b0)
        github.com/json-iterator/go@v1.1.11/reflect_optional.go:70 +0x104
github.com/json-iterator/go.(*sliceEncoder).Encode(0x4061fc0, 0x4028728, 0x416a4b0)
        github.com/json-iterator/go@v1.1.11/reflect_slice.go:38 +0x37c
github.com/json-iterator/go.(*OptionalEncoder).Encode(0x3ee8860, 0x3ee88a0, 0x416a4b0)
        github.com/json-iterator/go@v1.1.11/reflect_optional.go:70 +0x104
github.com/json-iterator/go.(*onePtrEncoder).Encode(0x3ee8868, 0x4028728, 0x416a4b0)
        github.com/json-iterator/go@v1.1.11/reflect.go:219 +0x8c
github.com/json-iterator/go.(*Stream).WriteVal(0x416a4b0, {0x1b4ac50, 0x4028728})
        github.com/json-iterator/go@v1.1.11/reflect.go:98 +0x1d4
github.com/json-iterator/go.(*frozenConfig).Marshal(0x3f9c540, {0x1b4ac50, 0x4028728})
        github.com/json-iterator/go@v1.1.11/config.go:299 +0xb4
github.com/containers/storage.(*layerStore).saveLayers(0x4028700)
        github.com/containers/storage@v1.36.0/layers.go:484 +0x150
github.com/containers/storage.(*layerStore).Save(0x4028700)
        github.com/containers/storage@v1.36.0/layers.go:467 +0xd8
github.com/containers/storage.(*layerStore).Put(0x4028700, {0x413a327, 0x40}, 0x0, {0x0, 0x0, 0x0}, {0x0, 0x0}, 0x0, ...)
        github.com/containers/storage@v1.36.0/layers.go:799 +0x1834
github.com/containers/storage.(*store).PutLayer(0x3cc80c0, {0x413a327, 0x40}, {0x0, 0x0}, {0x0, 0x0, 0x0}, {0x0, 0x0}, ...)
        github.com/containers/storage@v1.36.0/store.go:1057 +0xc38
github.com/containers/image/v5/storage.(*storageImageDestination).commitLayer(0x409e6e0, {0x1d5f65c, 0x3ec31a0}, {{{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, ...}, ...}, ...)
        github.com/containers/image/v5@v5.16.0/storage/storage_image.go:1010 +0xeb0
github.com/containers/image/v5/storage.(*storageImageDestination).queueOrCommit(0x409e6e0, {0x1d5f65c, 0x3ec31a0}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
        github.com/containers/image/v5@v5.16.0/storage/storage_image.go:837 +0x1dc
github.com/containers/image/v5/storage.(*storageImageDestination).PutBlobWithOptions(0x409e6e0, {0x1d5f65c, 0x3ec31a0}, {0x1d55dd4, 0x3eddd60}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, ...}, ...}, ...)
        github.com/containers/image/v5@v5.16.0/storage/storage_image.go:466 +0x11c
github.com/containers/image/v5/copy.(*copier).copyBlobFromStream(0x3eeea00, {0x1d5f65c, 0x3ec31a0}, {0x1d5779c, 0x3df4d20}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, ...}, ...}, ...)
        github.com/containers/image/v5@v5.16.0/copy/copy.go:1632 +0x1b34
github.com/containers/image/v5/copy.(*imageCopier).copyLayerFromStream(0x3ec2180, {0x1d5f65c, 0x3ec31a0}, {0x1d5779c, 0x3df4d20}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, ...}, ...}, ...)
        github.com/containers/image/v5@v5.16.0/copy/copy.go:1362 +0x348
github.com/containers/image/v5/copy.(*imageCopier).copyLayer.func3(0x3ec2180, 0x416ab40, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, {0x40e0640, ...}, ...}, ...)
        github.com/containers/image/v5@v5.16.0/copy/copy.go:1305 +0x19c
github.com/containers/image/v5/copy.(*imageCopier).copyLayer(0x3ec2180, {0x1d5f65c, 0x3ec31a0}, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, ...}, ...)
        github.com/containers/image/v5@v5.16.0/copy/copy.go:1329 +0x7b0
github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func1(0x0, {{0x413a5f0, 0x47}, 0x19aa38, {0x0, 0x0, 0x0}, 0x0, {0x40e0640, 0x31}, ...}, ...)
        github.com/containers/image/v5@v5.16.0/copy/copy.go:891 +0x338
created by github.com/containers/image/v5/copy.(*imageCopier).copyLayers.func2
        github.com/containers/image/v5@v5.16.0/copy/copy.go:928 +0x338
root@turris:~# 

[petrkr]

Including podman version/info

root@turris:~# podman version
Version:      3.4.1
API Version:  3.4.1
Go Version:   go1.18.9
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/arm

root@turris:~# podman info
host:
  arch: arm
  buildahVersion: 1.23.1
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - pids
  - rdma
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: Unknown
    path: /usr/bin/conmon
    version: 'conmon version 2.0.30, commit: '
  cpus: 2
  distribution:
    distribution: '"turrisos"'
    version: 6.2.1
  eventLogger: none
  hostname: turris
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.15.86
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 190943232
  memTotal: 2114068480
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/sbin/runc
    version: |-
      runc version 1.1.2
      spec: 1.0.2-dev
      go: go1.18.9
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: false
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 86h 2m 54.11s (Approximately 3.58 days)
plugins:
  log:
  - k8s-file
  - none
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /srv/containers/storage
  graphStatus:
    Build Version: 'Btrfs v5.11 '
    Library Version: "102"
  imageStore:
    number: 0
  runRoot: /tmp/run/containers/storage
  volumePath: /srv/containers/storage/volumes
version:
  APIVersion: 3.4.1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.18.9
  OsArch: linux/arm
  Version: 3.4.1

[dangowrt]

@oskarirauta

[oskarirauta]

©petrkr

Not sure, but didn't 3.4.1 fix this issue? Or some of the newer ones? My webserver (yes, my site's server runs on openwrt) - has been running with podman 3.4.4 ever since I updated to that version. There's weekly reboots on server, but it has nothing to do with podman.... Should update the whole system.

Btw. why is mine using cgroupv1 when you clearly have cgroupv2..?

Otherwise your setup seems just like mine that runs without issues except that is a bit older, and:

• cgroup version
• I have slirp ( but as we run these with root it doesn't have anything.. )
• my storage driver is extfs instead of btrfs, but that's propably more related on your choice of filesystem.

One thing though pops... You are using runc as oci runtime. Would you mind trying first with crun, as it's recommended for podman over runc.

I have never updated anything but master versions, current is running with 4.3.0 and I should be today, updating the build to 4.3.1. My main router is running with 4.3.0 as far as I know- without bigger problem, with basic alpine:latest and tmux keeps it up..

EDIT: Sorry, I seem to be instead running with 4.3.1 and I already have updated in master to 4.3.1 - I just got a notification about 4.4.0-rc3 so no update coming even to master at the moment, as I try to avoid updates to RC versions, unless their changelog reveals that they fix real issues, and atleast for rc2 there wasn't changelog listed at all.. Sorry, for misleading information, I just woke up.. But really, try switching to crun first.

[oskarirauta]

But try switch to crun first.

[petrkr]

I did not mentioned it, but I've triee both, runc and crun, same results

[oskarirauta]

Ok, so you need a updated version then. I can verify that 3.4.4 works fine.

[petrkr]

Ok, so you need a updated version then. I can verify that 3.4.4 works fine.

Can not update as package is not compiled. That is what I wrote on original issue.. if someone have working build environment, maybe can put and build 3.4.4 and I can test. But in 21.02 repo is still 3.4.1

[oskarirauta]

@petrkr

I have, but wrong arch- mine is x86-64.. There's though even newer version available, though I am not sure if other system depencies are filled-

Snapshots: podman_4.3.1-1

Double check the download path, there were alternatives for arm cortex a9 and I know nothing about turris. Other variants were neon and vfpv3-d16, most likely though, linked file is for your arch...

[petrkr]

# opkg install /root/podman_4.3.1-1_arm_cortex-a9.ipk 
Package podman (3.4.1-1) installed in root is up to date.
Collected errors:
 * pkg_hash_check_unresolved: cannot find dependency libncursesw for libreadline
 * pkg_hash_fetch_best_installation_candidate: Packages for libreadline found, but incompatible with the architectures configured

yes, there will be dependency hell as this package will be probably from 22/master branch.

[oskarirauta]

That is possible. I pushed it few weeks ago to master.

[paper42]

Updating to 3.4.7 helps, but I am still having issues:

root@turris:~# podman run -it --rm arm32v7/busybox
✔ docker.io/arm32v7/busybox:latest
Trying to pull docker.io/arm32v7/busybox:latest...
Getting image source signatures
Copying blob 46758452d3ee done  
Copying config 1d57ab16f6 done  
Writing manifest to image destination
Storing signatures
Error: OCI runtime error: prctl: Invalid argument

and this on any subsequent runs:

Error: container create failed (no logs from conmon): EOF

I didn't manage to build 4.3.1 on OpenWrt 21.02 even though gpg-error.pc is in build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/libgpg-error-1.45/ipkg-install/usr/lib/pkgconfig/gpg-error.pc:

# /build/build/staging_dir/host/bin/pkg-config --cflags  -- gpgme
Package gpg-error was not found in the pkg-config search path.
Perhaps you should add the directory containing `gpg-error.pc'
to the PKG_CONFIG_PATH environment variable
Package 'gpg-error', required by 'gpgme', not found
Package 'gpg-error', required by 'libassuan', not found

[petrkr]

Updating to 3.4.7 helps, but I am still having issues:

Can you please share that ipkg/opkg ? I will try to check this other stuff... But I hope it's not kernel config related

[paper42]

Updating to 3.4.7 helps, but I am still having issues:

Can you please share that ipkg/opkg ? I will try to check this other stuff... But I hope it's not kernel config related

podman_3.4.7-1_arm_cortex-a9_vfpv3-d16.zip GitHub didn't allow me to upload an .ipk file, so just change the extension from zip to ipk.

[petrkr]

Well. this is at-least interesting error message in syslog

Jan 31 14:42:03 turris modprobe: no module folders for kernel version 5.15.88 found

but what kind of kernel module or folder it looking for ?

EDIT: also syslog

Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/tmp/conmon-term.4GQEZ1} 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/proc/self/fd/12/attach} 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: terminal_ctrl_fd: 12 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: winsz read side: 15, winsz write side: 15 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: about to accept from console_socket_fd: 9 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: about to recvfd from connfd: 11 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <ninfo>: console = {.name = '(null)'; .fd = 0} 
Jan 31 14:44:20 turris : conmon 93689b3b1c74d1f21fb9 <nwarn>: Failed to get console terminal settings

[petrkr]

well, kernel module was just because OpenWRT updated to new kernel without reboot. So ignore that message.

[petrkr]

Just wonder if new version of "conmon" can fix it

[v2.0.32](https://github.com/containers/conmon/releases/tag/v2.0.32)
Bug Fixes
Avoid mainfd_std{in,out} sharing the same file descriptor.

in OpenWRT 21.02 is 2.0.30

[oskarirauta]

Latest conmon is 2.1.5 - available at snapshots.

[petrkr]

Latest conmon is 2.1.5 - available at snapshots.

Again, seems I can not use snapshot version in 21.02 OpenWRT

root@turris:~# conmon --version
Error relocating /usr/bin/conmon: __clock_gettime64: symbol not found
Error relocating /usr/bin/conmon: __localtime64_r: symbol not found
Error relocating /usr/bin/conmon: __dlsym_time64: symbol not found

[paper42]

conmon 2.1.5 and libseccomp 2.5.2 which was required for the build for OpenWrt 21.02: conmon_2.1.5-1_arm_cortex-a9_vfpv3-d16.zip libseccomp_2.5.2-1_arm_cortex-a9_vfpv3-d16.zip

I am still getting the Error: OCI runtime error: prctl: Invalid argument error

[petrkr]

There could be missing something in kernel. I check that debug message and it can not get terminal info, but which terminal? Container's or system (turris one).

I also tried all three runc, crun, uxc... actually uxc ends with "I/O Error" but since they missing lot of debug prints, I really do not know which part of 2000 lines C file ends with return -1

[oskarirauta]

remember to put uxc to different group, if I remember correctly, it doesn't support json like those 2 others- but nevertheless, uxc won't work anyway, I tested that sometime ago as well..