Cannot find libc on Fedora

[ekilmer]

I've been testing libtree on Fedora Linux distro and have been having issues trying to get the integration test to pass. Other binaries have a similar problem too:

$ libtree /bin/ls
ls
├── libselinux.so.1 not found: RPATH (empty) LD_LIBRARY_PATH (empty) RUNPATH (empty) /etc/ld.so.conf (empty)
├── libcap.so.2 not found: RPATH (empty) LD_LIBRARY_PATH (empty) RUNPATH (empty) /etc/ld.so.conf (empty)
└── libc.so.6 not found: RPATH (empty) LD_LIBRARY_PATH (empty) RUNPATH (empty) /etc/ld.so.conf (empty)

I believe the issue is due to the difference between Fedora and Ubuntu with the /etc/ld.so.conf.d directory, which is used by /etc/ld.so.conf.

On Fedora, there is no entry for libc:

$ docker run --rm fedora:32 ls -la /etc/ld.so.conf.d
total 8
drwxr-xr-x 2 root root 4096 Mar 21 01:08 .
drwxr-xr-x 1 root root 4096 May 30 21:48 ..

However, on Ubuntu (both 20.04 and 18.04), there are entries to find libc:

$ docker run --rm ubuntu:18.04 ls -al /etc/ld.so.conf.d
total 16
drwxr-xr-x 2 root root 4096 Mar 11 21:04 .
drwxr-xr-x 1 root root 4096 May 30 21:50 ..
-rw-r--r-- 1 root root   44 Jan 27  2016 libc.conf
-rw-r--r-- 1 root root  100 Apr 16  2018 x86_64-linux-gnu.conf

$ docker run --rm ubuntu:18.04 cat /etc/ld.so.conf.d/libc.conf
# libc default configuration
/usr/local/lib

$ docker run --rm ubuntu:18.04 cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf
# Multiarch support
/usr/local/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu

I think we could resolve this in a reasonable way by searching the output of the binary's loader (interpreter) output.

For example, in the integration test, we have the main binary (compiled on my Fedora 32 machine):

$ file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=d452578c100a02abb811e0e6cc8775d72aee6edb, for GNU/Linux 3.2.0, not stripped

We can use the interpreter (loader) path as our starting point and execute the following:

$ LD_DEBUG=libs /lib64/ld-linux-x86-64.so.2 --inhibit-cache ~/src/libtree/build/libtree
     55849:     find library=libstdc++.so.6 [0]; searching
     55849:      search path=/lib64/tls/haswell/x86_64:/lib64/tls/haswell:/lib64/tls/x86_64:/lib64/tls:/lib64/haswell/x86_64:/lib64/haswell:/lib64/x86_64:/lib64:/usr/lib64/tls/haswell/x86_64:/usr/lib64/tls/haswell:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/haswell/x86_64:/usr/lib64/haswell:/usr/lib64/x86_64:/usr/lib64                (system search path)
     55849:       trying file=/lib64/tls/haswell/x86_64/libstdc++.so.6
     55849:       trying file=/lib64/tls/haswell/libstdc++.so.6
     55849:       trying file=/lib64/tls/x86_64/libstdc++.so.6
     55849:       trying file=/lib64/tls/libstdc++.so.6
     55849:       trying file=/lib64/haswell/x86_64/libstdc++.so.6
     55849:       trying file=/lib64/haswell/libstdc++.so.6
     55849:       trying file=/lib64/x86_64/libstdc++.so.6
     55849:       trying file=/lib64/libstdc++.so.6
....
     55849:     initialize program: /home/ekilmer/src/libtree/build/libtree
     55849:
     55849:
     55849:     transferring control: /home/ekilmer/src/libtree/build/libtree
     55849:
Show the dependency tree of binaries and optionally bundle them into a single folder.
Usage:
  libtree [OPTION...] binary [more binaries...]

  -h, --help     Print usage
      --version  Print version info

 A. Locating libs options:
  -p, --path          Show the path of libraries instead of their SONAME
  -v, --verbose       Show the skipped libraries without their children
  -a, --all           Show the skipped libraries and their children
  -l, --ldconf arg    Path to custom ld.conf to test settings (default:
                      /etc/ld.so.conf)
  -s, --skip arg      Skip library and its dependencies from being deployed
                      or inspected
      --platform arg  Platform used for interpolation in rpaths (default:
                      x86_64)

 B. Copying libs options:
  -d, --destination arg  OPTIONAL: When a destination is set to a folder, all
                         binaries and their dependencies are copied over
      --strip            Call strip on binaries when deploying
      --chrpath          Call chrpath on binaries when deploying

     55849:
     55849:     calling fini: /home/ekilmer/src/libtree/build/libtree [0]
....

and parse the search path= line to add directories and search for the missing libraries.

NOTE Running ld-linux-x86-64.so.2 is equivalent to running the actual program, so we want to make sure the program exists and does a no-op. I would suggest just running our own libtree program to collect these paths. I was also thinking of maybe doing /bin/true, but that's not guaranteed to be there, and running the target program probably isn't a good idea.

Unfortunately, I haven't found a way to just have ld-linux-x86-64.so.2 (or the 32-bit version) print its search paths and do nothing else. That would be best case scenario.

One issue with this method would be if someone tries to run a 64-bit build of libtree on a 32-bit file. Using the target file's 32-bit loader would result in a mismatch with the 64-bit libtree. I'm not sure of a robust way to collect the loader's search paths without potentially causing damage, since using the loader is equivalent to running a program.

Here is a sample of 32-bit loader output on my 64-bit Fedora VM:

        96:      search path=/lib/tls/i686/sse2:/lib/tls/i686:/lib/tls/sse2:/lib/tls:/lib/i686/sse2:/lib/i686:/lib/sse2:/lib:/usr/lib/tls/i686/sse2:/usr/lib/tls/i686:/usr/lib/tls/sse2:/usr/lib/tls:/usr/lib/i686/sse2:/usr/lib/i686:/usr/lib/sse2:/usr/lib              (system search path)

I've also opened a Draft Pull Request (#26) that tests Fedora in CI, so we can test commits there.

[haampie]

Thanks for the research ;) I will have to check it out a bit later. Adding CI for Fedora is a great way to get this tested properly, thanks a lot already!

[haampie]

Unfortunately, I haven't found a way to just have ld-linux-x86-64.so.2 (or the 32-bit version) print its search paths and do nothing else. That would be best case scenario.

Actually this is how ldd works, and I would like to avoid it. Calling ldd ./x is pretty much equivalent to executing LD_TRACE_LOADED_OBJECTS=1 ./x, and it relies on the dynamic linker not to execute whenever this variable is set.

See https://catonmat.net/ldd-arbitrary-code-execution how this could be potentially exploited. A bit contrived, but still.

From man ld.so the last bullet point about search paths says

       o  In the default path /lib, and then /usr/lib.  (On some 64-bit archi‐
          tectures, the default paths for 64-bit shared  objects  are  /lib64,
          and then /usr/lib64.)  If the binary was linked with the -z nodeflib
          linker option, this step is skipped.

It does not mention to recursively search those folders btw. We have to see how Fedora determines those folders in the first place, maybe it is documented, or hard-coded in source, etc

[haampie]

Diving into the source of glibc it seems they use:

/* Get the generated information about the trusted directories.  Use
   an array of concatenated strings to avoid relocations.  See
   gen-trusted-dirs.awk.  */
#include "trusted-dirs.h"

static const char system_dirs[] = SYSTEM_DIRS;

and this awk script is used in the Makefile:

$(objpfx)trusted-dirs.st: Makefile $(..)Makeconfig
    $(make-target-directory)
    echo "$(subst :, ,$(default-rpath) $(user-defined-trusted-dirs))"    \
    | $(AWK) -f gen-trusted-dirs.awk > ${@:st=T};
    echo '#define DL_DST_LIB "$(notdir $(slibdir))"' >> ${@:st=T}
    $(move-if-change) ${@:st=T} ${@:st=h}
    touch $@

so I guess default-rpath and user-defined-trusted-dirs must have been set when running make to what you shared above. Can we see how Fedora calls make somewhere?

[ekilmer]

Very interesting! Thank you for taking a look at this and posting your research 😄

Can we see how Fedora calls make somewhere?

Are you talking about the verbose output of what is actually being executed while building libtree? I added -DCMAKE_VERBOSE_MAKEFILE=ON in the Fedora CI PR https://github.com/haampie/libtree/pull/26/checks?check_run_id=723851379#step:6:88

[haampie]

No I mean, how does Fedora build glibc, because if I understand correctly those folders

55849: search path=/lib64/tls/haswell/x86_64:/lib64/tls/haswell:/lib64/tls/x86_64:/lib64/tls:/lib64/haswell/x86_64:/lib64/haswell:/lib64/x86_64:/lib64:/usr/lib64/tls/haswell/x86_64:/usr/lib64/tls/haswell:/usr/lib64/tls/x86_64:/usr/lib64/tls:/usr/lib64/haswell/x86_64:/usr/lib64/haswell:/usr/lib64/x86_64:/usr/lib64 (system search path)

are constants baked into ld-linux-x86-64.so.2. Where are they listed though

[haampie]

Closed by #26 for now. In case we need more exotic search paths, we have to think of a way to extract them from the dynamic linker.