Can't run podman container: open `proc`: no such file or directory: oci runtime command not found error

nightuser commented 4 years ago

OS: ArchLinux armv7h fuse-overlayfs 0.7.8 podman 1.8.2

With fuse-overlayfs driver I get the following error when creating a podman container:

$ podman run --log-level debug ubuntu
DEBU[0000] Reading configuration file "/home/nightuser/.config/containers/libpod.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/nightuser/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/nightuser/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/nightuser/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/nightuser/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] No store required. Not opening container store. 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] running as rootless                          
DEBU[0000] Reading configuration file "/home/nightuser/.config/containers/libpod.conf" 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/nightuser/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/nightuser/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/nightuser/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/nightuser/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] parsed reference into "[overlay@/home/nightuser/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/ubuntu:latest" 
DEBU[0000] parsed reference into "[overlay@/home/nightuser/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d70ce34baa497b9a4aad5220ead98587ee6ea5b10b08615ebfca2839107689eb" 
DEBU[0000] exporting opaque data as blob "sha256:d70ce34baa497b9a4aad5220ead98587ee6ea5b10b08615ebfca2839107689eb" 
DEBU[0000] Using slirp4netns netmode                    
DEBU[0000] No hostname set; container's hostname will default to runtime default 
DEBU[0000] Loading default seccomp profile              
DEBU[0000] created OCI spec and options for new container 
DEBU[0000] Allocated lock 0 for container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b 
DEBU[0000] parsed reference into "[overlay@/home/nightuser/.local/share/containers/storage+/run/user/1000/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@d70ce34baa497b9a4aad5220ead98587ee6ea5b10b08615ebfca2839107689eb" 
DEBU[0000] exporting opaque data as blob "sha256:d70ce34baa497b9a4aad5220ead98587ee6ea5b10b08615ebfca2839107689eb" 
DEBU[0000] created container "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" 
DEBU[0000] container "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" has work directory "/home/nightuser/.local/share/containers/storage/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata" 
DEBU[0000] container "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" has run directory "/run/user/1000/containers/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata" 
DEBU[0000] New container created "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" 
DEBU[0000] container "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" has CgroupParent "user.slice/user-1000.slice/user@1000.service/user.slice/libpod-fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b.scope" 
DEBU[0000] Not attaching to stdin                       
DEBU[0000] overlay: mount_data=lowerdir=/home/nightuser/.local/share/containers/storage/overlay/l/EDSEWEYLEOERADI6MIKBL2A5KT:/home/nightuser/.local/share/containers/storage/overlay/l/G5DWIQSHCRCIHVIMTQCREFWM2S:/home/nightuser/.local/share/containers/storage/overlay/l/7YNSE2WGMNWEHL6LGZFJQFVWAY:/home/nightuser/.local/share/containers/storage/overlay/l/7532T2KP6REUP7XZOGWS74K2Q7,upperdir=/home/nightuser/.local/share/containers/storage/overlay/edcfe9f348dc16e530fd2a5ae01f007869a0182a511e6bac75cb2c076f4df13b/diff,workdir=/home/nightuser/.local/share/containers/storage/overlay/edcfe9f348dc16e530fd2a5ae01f007869a0182a511e6bac75cb2c076f4df13b/work 
DEBU[0000] Made network namespace at /run/user/1000/netns/cni-dbadb58c-fe72-aebd-a939-61939bd6633b for container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b 
DEBU[0000] mounted container "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" at "/home/nightuser/.local/share/containers/storage/overlay/edcfe9f348dc16e530fd2a5ae01f007869a0182a511e6bac75cb2c076f4df13b/merged" 
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-dbadb58c-fe72-aebd-a939-61939bd6633b tap0 
DEBU[0000] Created root filesystem for container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b at /home/nightuser/.local/share/containers/storage/overlay/edcfe9f348dc16e530fd2a5ae01f007869a0182a511e6bac75cb2c076f4df13b/merged 
INFO[0000] No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4] 
INFO[0000] IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844] 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b to user.slice:libpod:fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b 
DEBU[0000] Created OCI spec for container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b at /home/nightuser/.local/share/containers/storage/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b -u fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b -r /usr/bin/crun -b /home/nightuser/.local/share/containers/storage/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata -p /run/user/1000/containers/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata/pidfile -l k8s-file:/home/nightuser/.local/share/containers/storage/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/nightuser/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg debug --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b.scope 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b 
DEBU[0000] Tearing down network namespace at /run/user/1000/netns/cni-dbadb58c-fe72-aebd-a939-61939bd6633b for container fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b 
DEBU[0000] unmounted container "fecd16f648432dc9bfb62683de25da054887828488d36e36a8689ad10e5f8e4b" 
DEBU[0001] ExitCode msg: "open `proc`: no such file or directory: oci runtime command not found error" 
ERRO[0001] open `proc`: No such file or directory: OCI runtime command not found error

This happens with both crun and runc (although, with runc the error message it a bit different, but the gist is the same). The configuration with vfs works without the issue.

Update: I tried testing fuse-overlayfs binary with unionmount-testsuite and got the following:

$ sudo unshare -m ./run --ov --fuse=fuse-overlayfs --xdev -v
***
*** ./run --ov --ts=0 open-plain
***
os.lstat(/lower/a)
os.lstat(/upper)
os.lstat(/mnt/a)
Traceback (most recent call last):
  File "./run", line 313, in <module>
    mount_union(ctx)
  File "/home/nightuser/dev/unionmount-testsuite/mount_union.py", line 46, in mount_union
    ctx.note_upper_fs(upper_mntroot, testdir, union_mntroot + "/f")
  File "/home/nightuser/dev/unionmount-testsuite/context.py", line 295, in note_upper_fs
    self.__upper_dir_fs = self.get_dev_id(dirpath)
  File "/home/nightuser/dev/unionmount-testsuite/context.py", line 268, in get_dev_id
    return self.lstat_file(path).st_dev
  File "/home/nightuser/dev/unionmount-testsuite/context.py", line 261, in lstat_file
    return os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/a'

If I try to ls /mnt, I get ls: reading directory '/mnt': Input/output error. At the same time /lower/a exists. I don't really now much about overlay filesystems and similar stuff, so I can only provide some additional debug info (or possibly an affected container / ssh access to my board).

At the same time, on my amd64 ArchLinux laptop it passes the tests.

nightuser commented 4 years ago

Sorry for the double-post, but the first one is kind of long.

The easiest way to reproduce the issue is using a podman container.

First, make sure that you have qemu-user-static and binfmt set up correctly. Then use the following Dockerfile:

FROM arm32v7/ubuntu:bionic

RUN apt-get update && apt-get install -y \
      automake \
      autotools-dev \
      gcc \
      g++ \
      git \
      udev \
      pkg-config \
      ninja-build \
      meson \
      wget \
      fuse
RUN (cd /; git clone https://github.com/amir73il/unionmount-testsuite.git)
RUN (cd /; git clone https://github.com/containers/fuse-overlayfs.git)
RUN (cd /; wget https://github.com/libfuse/libfuse/releases/download/fuse-3.6.2/fuse-3.6.2.tar.xz; tar xf fuse-3.6.2.tar.xz; cd fuse-3.6.2; mkdir build; cd build; meson .. --prefix /usr -D examples=false && ninja && ninja install)
RUN (cd /fuse-overlayfs; ./autogen.sh; ./configure; make -j $(nproc); make install; cp fuse-overlayfs /sbin)
RUN mkdir -p /lower /upper /mnt
WORKDIR /unionmount-testsuite
CMD unshare -m ./run --ov --fuse=fuse-overlayfs --xdev -v

And create a container (as root):

podman build --tag fuse-overlayfs-arm -f Dockerfile
podman run --privileged fuse-overlayfs-arm

MastaG commented 4 years ago

Same issue here on Fedora 32 armv7.

crun-0.13-1.fc32.armv7hl
fuse-2.9.9-9.fc32.armv7hl
fuse3-3.9.1-1.fc32.armv7hl
fuse3-libs-3.9.1-1.fc32.armv7hl
fuse-common-3.9.1-1.fc32.armv7hl
fuse-libs-2.9.9-9.fc32.armv7hl
fuse-overlayfs-0.7.8-1.fc32.armv7hl
libgpod-0.8.3-32.fc32.armv7hl
libslirp-4.2.0-1.fc32.armv7hl
podman-1.8.2-2.fc32.armv7hl
podman-plugins-1.8.2-2.fc32.armv7hl
slirp4netns-0.4.3-6.0.dev.gita8414d1.fc32.armv7hl

Output of podman version:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.14
OS/Arch:            linux/arm

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.14
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.14-1.fc32.armv7hl
    path: /usr/bin/conmon
    version: 'conmon version 2.0.14, commit: a563978c8cac161fa1adf986291f7497e4b974f2'
  Distribution:
    distribution: fedora
    version: "32"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 1304940544
  MemTotal: 2079457280
  OCIRuntime:
    name: crun
    package: crun-0.13-1.fc32.armv7hl
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 2147479552
  SwapTotal: 2147479552
  arch: arm
  cpus: 8
  eventlogger: journald
  hostname: odroidxu4
  kernel: 5.4.3-odroid-xu3_and_4+
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.3-6.0.dev.gita8414d1.fc32.armv7hl
    Version: |-
      slirp4netns version 0.4.3+dev
      commit: a8414d1d1629f6f7a93b60b55e183a93d10d9a1c
  uptime: 26m 41.5s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /home/odroid/.config/containers/storage.conf
  ContainerStore:
    number: 18
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.8-1.fc32.armv7hl
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 0.7.8
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  GraphRoot: /home/odroid/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: f2fs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000/containers
  VolumePath: /home/odroid/.local/share/containers/storage/volumes

Kernel: 5.4.3

# CONFIG_OF_OVERLAY is not set
CONFIG_FUSE_FS=m
CONFIG_OVERLAY_FS=m
# CONFIG_OVERLAY_FS_REDIRECT_DIR is not set
CONFIG_OVERLAY_FS_REDIRECT_ALWAYS_FOLLOW=y
# CONFIG_OVERLAY_FS_INDEX is not set
# CONFIG_OVERLAY_FS_XINO_AUTO is not set
# CONFIG_OVERLAY_FS_METACOPY is not set

But it only happens when trying to run rootless containers. Running containers as root works fine.

Here's the output of:

$ LANG=C podman unshare strace -f -Z -e open,openat podman run --rm -ti hello-world

open("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = -1 ENOENT (No such file or directory)
strace: Process 1962 attached
strace: Process 1963 attached
strace: Process 1964 attached
strace: Process 1965 attached
strace: Process 1966 attached
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 1967 attached
strace: Process 1968 attached
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 1969 attached
[pid  1969] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1969] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1969] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1969] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1965] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 1970 attached
strace: Process 1971 attached
[pid  1969] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1968] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1968] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] openat(AT_FDCWD, "/sys/fs/cgroup/cpuset.cpus", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1961] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-iptables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1961] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-ip6tables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] openat(AT_FDCWD, "/sys/fs/cgroup/cpuset.cpus", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1961] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-iptables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1961] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-ip6tables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 1972 attached
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1972] openat(AT_FDCWD, "/usr/lib/charset.alias", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid  1972] +++ exited with 0 +++
[pid  1967] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1972, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 1973 attached
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1973] openat(AT_FDCWD, "/usr/lib/charset.alias", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid  1973] +++ exited with 0 +++
[pid  1967] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1973, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] openat(AT_FDCWD, "/home/odroid/.local/share/containers/storage/overlay/aa00a4b32179427505facdf4348f7b8ebefec5f8b48e76595e9055e119f56321/lower", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] openat(AT_FDCWD, "/home/odroid/.local/share/containers/storage/overlay/aa00a4b32179427505facdf4348f7b8ebefec5f8b48e76595e9055e119f56321/lower", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1965] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1963] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1965] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 1975 attached
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1969] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1967] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1975] +++ exited with 0 +++
                                 [pid  1971] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                      [pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                                                                                                           strace: Process 1976 attached
                                                                                                                                                                                                                                        strace: Process 1977 attached
                   [pid  1971] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                        [pid  1977] +++ exited with 0 +++
                                                                                                                                         [pid  1961] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1977, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
                     [pid  1971] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                          [pid  1976] openat(AT_FDCWD, "/usr/libexec/fuse-overlayfs", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
                  strace: Process 1978 attached
                                               strace: Process 1979 attached
                                                                            [pid  1976] +++ exited with 0 +++
                                                                                                             [pid  1961] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1976, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
                                                                                                                                                                                                                                           [pid  1971] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                              strace: Process 1981 attached
                                                                                                           [pid  1981] +++ exited with 0 +++
                                                                                                                                            [pid  1978] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1981, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
                        [pid  1970] openat(AT_FDCWD, "/home/odroid/.local/share/containers/storage/overlay/85ef3eb7e51903141531d77d16ee7458c094e2a399048e190d6c0fcdb5903c83/merged/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
                     [pid  1970] openat(AT_FDCWD, "/home/odroid/.local/share/containers/storage/overlay/85ef3eb7e51903141531d77d16ee7458c094e2a399048e190d6c0fcdb5903c83/merged/etc/group", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
                 [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                      strace: Process 1982 attached
                                                                                                                                   [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                                                                                                                        strace: Process 1983 attached
   [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                        [pid  1983] openat(AT_FDCWD, "/usr/lib/charset.alias", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
                                                                                                                                                                                                            strace: Process 1984 attached
                                                                                                                                                                                                                                         [pid  1983] +++ exited with 0 +++
                        [pid  1961] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1983, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
                                                                                                                                                      strace: Process 1985 attached
                                                                                                                                                                                   strace: Process 1986 attached
                                                                                                                                                                                                                strace: Process 1987 attached
                                                                                                                                                                                                                                             strace: Process 1988 attached
                        [pid  1987] +++ exited with 0 +++
                                                         [pid  1985] openat(AT_FDCWD, "/sys/fs/cgroup/cgroup.subtree_control", O_WRONLY|O_CREAT, 0700) = -1 EACCES (Permission denied)
                                                                                                                                                                                      [pid  1985] openat(AT_FDCWD, "/sys/fs/cgroup/user.slice/cgroup.subtree_control", O_WRONLY|O_CREAT, 0700) = -1 EACCES (Permission denied)
                                                                            [pid  1985] openat(AT_FDCWD, "/sys/fs/cgroup/user.slice/user-1000.slice/cgroup.subtree_control", O_WRONLY|O_CREAT, 0700) = -1 EACCES (Permission denied)
                                                                                                                                                                                                                                    [pid  1979] openat(3, "proc", O_RDONLY|O_NONBLOCK|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
                                                                                       [pid  1979] openat(4, "proc/.opq", O_RDONLY|O_NONBLOCK|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
                                                                                                                                                                                                 [pid  1988] openat(9, "proc", O_RDONLY|O_CLOEXEC|O_PATH) = -1 ENOENT (No such file or directory)
                                               [pid  1985] openat(AT_FDCWD, "/run/user/1000/crun/2d77b22b64221c19e77418c8e8b06d77232a478f0a87fbd6e5df8b8b1804d997/status", O_RDONLY) = -1 ENOENT (No such file or directory)
                                                                                                                                                                                                                            [pid  1988] +++ exited with 1 +++
           [pid  1984] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1988, si_uid=0, si_status=1, si_utime=0, si_stime=0} ---
                                                                                                                                         [pid  1985] +++ exited with 1 +++
                                                                                                                                                                          [pid  1984] --- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_TKILL, si_pid=1984, si_uid=0} ---
               [pid  1984] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1987, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
                                                                                                                                             [pid  1984] --- SIGUSR1 {si_signo=SIGUSR1, si_code=SI_TKILL, si_pid=1984, si_uid=0} ---
                                                                                                                                                                                                                                    strace: Process 1989 attached
               [pid  1961] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                    [pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                                                                                         strace: Process 1990 attached
                                                                                                                                                                                                                      [pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                         [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                                              [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                                                                                                                                   [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                      [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                                                           strace: Process 1991 attached
                                                                                                                                                                                        [pid  1991] openat(AT_FDCWD, "/run/user/1000/crun/2d77b22b64221c19e77418c8e8b06d77232a478f0a87fbd6e5df8b8b1804d997/status", O_RDONLY) = -1 ENOENT (No such file or directory)
                                                                                                                   [pid  1991] openat(3, "2d77b22b64221c19e77418c8e8b06d77232a478f0a87fbd6e5df8b8b1804d997", O_RDONLY|O_DIRECTORY) = -1 ENOENT (No such file or directory)
                        [pid  1991] +++ exited with 0 +++
                                                         [pid  1970] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1991, si_uid=0, si_status=0, si_utime=0, si_stime=3} ---
                                                                                                                                                                                       [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                          [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                               [pid  1989] open("/sys/kernel/mm/transparent_hugepage/hpage_pmd_size", O_RDONLY) = -1 ENOENT (No such file or directory)
                                                                                                                                                                                                                                       strace: Process 1993 attached
                  strace: Process 1994 attached
                                               strace: Process 1995 attached
                                                                            [pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
                                                                                                                                                                 [pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
    strace: Process 1996 attached
                                 strace: Process 1997 attached
                                                              strace: Process 1998 attached
                                                                                           strace: Process 1999 attached
                                                                                                                        [pid  1993] +++ exited with 0 +++
                                                                                                                                                         [pid  1970] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1993, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
                                     [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
                                                                                                                          [pid  1979] +++ exited with 0 +++
                                                                                                                                                           [pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
                                                                                                                                                                                                                                                strace: Process 2000 attached
                           [pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1968] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1965] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1964] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1968] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
strace: Process 2002 attached
strace: Process 2003 attached
[pid  2002] openat(AT_FDCWD, "/run/user/1000/crun/2d77b22b64221c19e77418c8e8b06d77232a478f0a87fbd6e5df8b8b1804d997/status", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid  2002] openat(3, "2d77b22b64221c19e77418c8e8b06d77232a478f0a87fbd6e5df8b8b1804d997", O_RDONLY|O_DIRECTORY) = -1 ENOENT (No such file or directory)
[pid  2002] +++ exited with 0 +++
[pid  1970] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2002, si_uid=0, si_status=0, si_utime=0, si_stime=3} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
[pid  1970] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1961, si_uid=0} ---
Error: open `proc`: No such file or directory: OCI runtime command not found error
[pid  1966] +++ exited with 127 +++
[pid  1967] +++ exited with 127 +++
[pid  1964] +++ exited with 127 +++
[pid  1969] +++ exited with 127 +++
[pid  1990] +++ exited with 127 +++
[pid  1982] +++ exited with 127 +++
[pid  1971] +++ exited with 127 +++
[pid  1970] +++ exited with 127 +++
[pid  1968] +++ exited with 127 +++
[pid  1965] +++ exited with 127 +++
[pid  1963] +++ exited with 127 +++
[pid  1962] +++ exited with 127 +++
[pid  1961] +++ exited with 127 +++
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1997] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1997] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  2000] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1997] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
strace: Process 2004 attached
strace: Process 2005 attached
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] openat(AT_FDCWD, "/sys/fs/cgroup/cpuset.cpus", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1989] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-iptables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1989] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-ip6tables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] openat(AT_FDCWD, "/sys/fs/cgroup/cpuset.cpus", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1989] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-iptables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1989] openat(AT_FDCWD, "/proc/sys/net/bridge/bridge-nf-call-ip6tables", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1989] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
strace: Process 2006 attached
[pid  2006] openat(AT_FDCWD, "/usr/lib/charset.alias", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid  2006] +++ exited with 0 +++
[pid  1996] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2006, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
[pid  1996] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1996] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
strace: Process 2007 attached
[pid  1996] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  2007] openat(AT_FDCWD, "/usr/lib/charset.alias", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
[pid  2007] +++ exited with 0 +++
[pid  2005] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2007, si_uid=0, si_status=0, si_utime=0, si_stime=1} ---
[pid  2005] --- SIGURG {si_signo=SIGURG, si_code=SI_TKILL, si_pid=1989, si_uid=0} ---
[pid  1994] ????( <unfinished ...>
[pid  2005] +++ exited with 125 +++
[pid  1999] +++ exited with 125 +++
[pid  2003] +++ exited with 125 +++
[pid  2004] +++ exited with 125 +++
[pid  2000] +++ exited with 125 +++
[pid  1998] +++ exited with 125 +++
[pid  1997] +++ exited with 125 +++
[pid  1996] +++ exited with 125 +++
[pid  1995] +++ exited with 125 +++
[pid  1994] +++ exited with 125 +++
[pid  1989] +++ exited with 125 +++
[pid  1984] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=1989, si_uid=0, si_status=125, si_utime=12, si_stime=48} ---
[pid  1986] +++ exited with 0 +++
[pid  1984] +++ exited with 0 +++
+++ exited with 0 +++
Error: exit status 127

The exact same command works on Fedora 32 x86_64 with the exact same package versions. Messages that stand out are:

openat(3, "proc", O_RDONLY|O_NONBLOCK|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
openat(4, "proc/.opq", O_RDONLY|O_NONBLOCK|O_NOFOLLOW) = -1 ENOENT (No such file or directory)
openat(9, "proc", O_RDONLY|O_CLOEXEC|O_PATH) = -1 ENOENT (No such file or directory)
Error: open `proc`: No such file or directory: OCI runtime command not found error

I'm not getting any of the above on x86_64. Any ideas?

MastaG commented 4 years ago

I tried uninstalling fuse-overlayfs from the repository and compiling the master branch. But still the same issue when trying to use it with rootless containers. Something funky must be going on armv7.

MastaG commented 4 years ago

@nightuser Which filesystem are you using for your rootfs? I'm using f2fs.

When I look at main.c

#define OPAQUE_WHITEOUT ".wh..wh..opq"
...
static int
set_fd_opaque (int fd)
{
  cleanup_close int opq_whiteout_fd = -1;
  int ret;

  ret = fsetxattr (fd, PRIVILEGED_OPAQUE_XATTR, "y", 1, 0);
  if (ret < 0)
    {
      if (errno == ENOTSUP)
        goto create_opq_whiteout;
      if (errno != EPERM || (fsetxattr (fd, OPAQUE_XATTR, "y", 1, 0) < 0 && errno != ENOTSUP))
          return -1;
    }
 create_opq_whiteout:
  opq_whiteout_fd = TEMP_FAILURE_RETRY (openat (fd, OPAQUE_WHITEOUT, O_CREAT|O_WRONLY|O_NONBLOCK, 0700));
  return (opq_whiteout_fd >= 0 || ret == 0) ? 0 : -1;
}

This seems to be the failing block.. perhaps it only happens on a certain filesystem?

nightuser commented 4 years ago

@MastaG ext4. It works with the same fs on x86-64.

MastaG commented 4 years ago

Thank you @nightuser Then I guess somebody with more knowledge on the code-base should have a look, as I'm not a developer.

MastaG commented 4 years ago

@giuseppe You've added support for opaque whiteout in this PR: https://github.com/containers/fuse-overlayfs/pull/68/

Any idea why it would fail on armv7 ? Could it be kernel related?

rhatdan commented 4 years ago

@giuseppe PTAL

giuseppe commented 4 years ago

I don't think it depends on that PR.

I think it could be related to some other portability issue.

I've no easy access right now to armv7 to try it out.

@MastaG could you try if other FUSE file systems (e.g. sshfs) work fine on your system?

MastaG commented 4 years ago

@giuseppe if you send me an email to mastag @ gmail dot com I'll grant you access to my odroid machine.

sshfs 3.7.0-3.fc32 works fine. I just mounted the home directory of my x86_64 machine.

nightuser commented 4 years ago

@giuseppe @MastaG I used git bisect and the problem first appeared in the commit c2c2ac5b82fb59322da227d196214b4a58ede634

giuseppe commented 4 years ago

thanks for your help.

I confirm the regression was introduced by that commit.

I've tested the PR on the machine @MastaG provided me. I've also left a fixed version at /usr/bin/fuse-overlayfs.

@nightuser could you take a look as well?

I'll cut a new release once it is confirmed to work

nightuser commented 4 years ago

@giuseppe Yes. It works on my machine too. Thank you!

nightuser commented 4 years ago

@giuseppe excuse me for an unrelated question, but how did you find out what caused the issue? I tried to fix it myself, but to no avail.

giuseppe commented 4 years ago

First I've checked what calls went into fuse-overlayfs.

For that, I've used fuse-overlayfs -f --debug .....

The -f prevent the process from detaching and the --debug prints what operations are done.

Since there was only opendir/readdir done, I've attached the debugger to fuse-overlayfs and debugged the ovl_do_readdir. Then I've realized it was trying to read some garbage from the direct.c file.

nightuser commented 4 years ago

@giuseppe oh, thanks. I figured out the flags, but tried to analyze strace output instead of using gdb.

MastaG commented 4 years ago

Thank you @giuseppe sorry I couldn't reply earlier. The ODroid community will love it, now rootless containers are working again!

rhatdan commented 4 years ago

@MastaG are you using Podman?

MastaG commented 4 years ago

@rhatdan Yes I'm running podman on Fedora on all of my devices (servers and vm's). But it wasn't working any longer on my armv7 devices (raspberry pi and odroid). But now it does again :)

This means I can now release a F32 image for odroid xu3/xu4 and later odroid n2.

rhatdan commented 4 years ago

Nice.

containers / fuse-overlayfs

Can't run podman container: open `proc`: no such file or directory: oci runtime command not found error #197