Symbol lookup error for add_elem_to_mount_list when using lxc-usernsexec on F30 (unpriviledged containers)

[vrubiolo]

Hi there,

First, thanks for making a Copr repository available for lxc3 on Fedora! This is a great way to use it w/ minimal fiddling.

I am currently trying to use it for unprivileged containers on Fedora30 and this fails with a symbol lookup error (running as root works fine):

$ lxc-create -n mycontainer -t download -- --dist fedora --release 29 --arch amd64 
lxc-create: mycontainer: conf.c: chown_mapped_root: 3182 lxc-usernsexec failed: lxc-usernsexec: symbol lookup error: lxc-usernsexec: undefined symbol: add_elem_to_mount_list
lxc-create: mycontainer: tools/lxc_create.c: main: 331 Failed to create container mycontainer

I think I have the setup right:

$ id
uid=1000(vincent) gid=1000(vincent) groups=1000(vincent),10(wheel),1001(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

$ cat /etc/subgid
vincent:100000:65536

$ cat /etc/subuid
vincent:100000:65536

$ cat ~/.config/lxc/default.conf 
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 100000 65536
lxc.idmap = g 0 100000 65536

And yet this still fails. Looking at the lxc source code, I see it's when chown_mapped_root does an execve of the binary that this ends up failing but ldd shows all libs as being resolved correctly.

Would you have some pointers? Below is my system info.

Thanks much!

$ cat /etc/redhat-release 
Fedora release 30 (Thirty)

$ which lxc-usernsexec |xargs rpm -qf
lxc-3.2.1-0.3.fc30.x86_64

$ ldd /usr/bin/lxc-usernsexec
    linux-vdso.so.1 (0x00007ffddf719000)
    liblxc.so.1 => /lib64/liblxc.so.1 (0x00007f80dd253000)
    libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f80dcf72000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f80dcf6d000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f80dcf4c000)
    libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f80dcf32000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f80dcd6c000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f80dcd3d000)
    libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007f80dccf4000)
    libcap.so.2 => /lib64/libcap.so.2 (0x00007f80dcced000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f80dd35b000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f80dccd3000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f80dcccd000)
    libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f80dcc3f000)

[ganto]

Thanks for your interest in my repository. I just tried to reproduce your issue on a Fedora 29 that I had available but didn't manage to.

Just some quick questions:

• Do you have SELinux on enforcing? Can you check if there are some alerts related to container creation?
• Did this error only appear after an update or is this a fresh installation?
• Can this error be reproduced after a reboot?
• Can you show me the output of rpm -qf /lib64/liblxc.so.1?

[vrubiolo]

Hi Reto,

Thanks for your reply and the associated guidance. I have started to follow/check the steps you were advising me to do and unfortunately I cannot reproduce the issue anymore. Since I reported it, the following has happened:

1. I checked and finished the configuration to allow networking of priviledged containers (lxcbr0-based). At the time, priviledged containers could be started but networking would fail. This is now fixed.
2. There were packages upgrades on the distro (not sure which ones, this is the automated dnf update).

For the record, here is the information you requested:

• selinux is enforced:

  $ sestatus 
  SELinux status:                 enabled
  SELinuxfs mount:                /sys/fs/selinux
  SELinux root directory:         /etc/selinux
  Loaded policy name:             targeted
  Current mode:                   enforcing
  Mode from config file:          enforcing
  Policy MLS status:              enabled
  Policy deny_unknown status:     allowed
  Memory protection checking:     actual (secure)
  Max kernel policy version:      31

• this was a fresh install from a month ago but I cannot be sure there weren't updates since then (I did take the updates today though, cf above)

• after the updates, the machine did reboot. Not sure if this is what caused the issue to not be reproducible.

• LXC library:

  $ rpm -qf /lib64/liblxc.so.1
  lxc-libs-3.2.1-0.3.fc30.x86_64

  Sorry for the noise here, I think we can close this issue. Let me know if you want me to provide additional information here (as others stumbling upon the same issue might find it worthwhile)

[ganto]

Thanks for the reply. I'm glad it's working for you now.

Actually I don't understand what went wrong. My gut feeling somehow pointed into the direction of still having an old version of the library loaded into the memory that's why I asked for a reboot or something blocking access to the symbol table that's why I asked for the SELinux logs. Not sure if any of this makes sense.

[vrubiolo]

Thanks for the followup and the associated help. Yes, I think your gut feeling was probably the right one, given the symbol-related error we got and that ldd did correctly solve the associated shared libraries.