search

haampie / libtree

ldd as a tree
MIT License
2.63k stars 60 forks source link

Different values to `ldd` in Gentoo Prefix #60

Closed ocaisa closed 2 years ago

ocaisa commented 2 years ago

I was trying to use libtree for some testing under Gentoo Prefix. I know we may need to patch CUDA to look in the prefix but I was surprised that ldd and lddtree don't return the same information:

[ocaisa@gpunode1 release]$ ldd ./deviceQuery
        linux-vdso.so.1 (0x00007ffd78ed6000)
        librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x0000153ce5eac000)
        libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x0000153ce5e8c000)
        libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x0000153ce5e86000)
        libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x0000153ce5c71000)
        libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x0000153ce5b30000)
        libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x0000153ce5b13000)
        libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x0000153ce5958000)
        /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x0000153ce5ec3000)
[ocaisa@gpunode1 release]$ lddtree ./deviceQuery
deviceQuery => ./deviceQuery (interpreter => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2)
    librt.so.1 => /lib64/librt.so.1
    libpthread.so.0 => /lib64/libpthread.so.0
    libdl.so.2 => /lib64/libdl.so.2
    libstdc++.so.6 => /lib64/libstdc++.so.6
    libm.so.6 => /lib64/libm.so.6
    libgcc_s.so.1 => /lib64/libgcc_s.so.1
    libc.so.6 => /lib64/libc.so.6
    ld-linux-x86-64.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
[ocaisa@gpunode1 release]$ which ldd
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/bin/ldd

Which is correct here? Is it that libtree is using the system linker rather than the one specified in the executable?

haampie commented 2 years ago

lddtree != libtree, seems like you're opening an issue in the wrong repo?

ocaisa commented 2 years ago

Argh, idiot, sorry, I do have libtree installed but was running the wrong command!

ocaisa commented 2 years ago

libtree not so helpful here unfortunately:

[ocaisa@gpunode1 release]$ ldd deviceQuery
        linux-vdso.so.1 (0x00007ffc0d994000)
        librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x0000148061437000)
        libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x0000148061417000)
        libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x0000148061411000)
        libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x00001480611fc000)
        libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x00001480610bb000)
        libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x000014806109e000)
        libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x0000148060ee3000)
        /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014806144e000)
[ocaisa@gpunode1 release]$ libtree deviceQuery
deviceQuery

[ocaisa@gpunode1 release]$
haampie commented 2 years ago

haha, but did libtree give you the right results on Gentoo Prefix? Try with -p and preferably 3.0.0-rc version :) I don't know how Gentoo Prefix manages their default search paths if they put the runtime linker in a prefix too

haampie commented 2 years ago

Sorry, use libtree -vp ..., as libtree doesn't show c / c++ libs by default to reduce verbosity.

ocaisa commented 2 years ago

Ok, things are starting to at least look similar now:

[ocaisa@gnode1 release]$ libtree --version
3.0.0-rc7
[ocaisa@gnode1 release]$ libtree -vp ./deviceQuery
./deviceQuery 
├── /lib64/librt.so.1 [default path]
│   ├── /lib64/libpthread.so.0 [default path]
│   └── /lib64/libc.so.6 [default path]
├── /lib64/ld-linux-x86-64.so.2 [default path]
├── /lib64/libc.so.6 [default path]
├── /lib64/libgcc_s.so.1 [default path]
├── /lib64/libm.so.6 [default path]
├── /lib64/libstdc++.so.6 [default path]
├── /lib64/libdl.so.2 [default path]
└── /lib64/libpthread.so.0 [default path]
[ocaisa@gnode1 release]$ ldd deviceQuery
        linux-vdso.so.1 (0x000014ba62c49000)
        librt.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 (0x000014ba62c2e000)
        libpthread.so.0 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 (0x000014ba62c0e000)
        libdl.so.2 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 (0x000014ba62c08000)
        libstdc++.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 (0x000014ba629f3000)
        libm.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 (0x000014ba628b2000)
        libgcc_s.so.1 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 (0x000014ba62895000)
        libc.so.6 => /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 (0x000014ba626da000)
        /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x000014ba62c4b000)
haampie commented 2 years ago

Right. So the prefix is missing. Is /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 a wrapper around the actual runtime linker that sets the gentoo prefix dir as a default library search path?

ocaisa commented 2 years ago

Our Gentoo Prefix is using it's own glibc with these paths baked in

[ocaisa@gpunode1 release]$ ls -l /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2
lrwxrwxrwx 1 cvmfs cvmfs 10 Jun 22 13:22 /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 -> ld-2.33.so

I think that answers your question but I'm not an expert here (@bedroge knows a lot more).

haampie commented 2 years ago

Thanks, would be good to know how Gentoo Prefix sets its default library search directories. libtree currently hard-codes them.

I was thinking to add a --sys-root flag, which is useful for cross compilation, but it could maybe also help with gentoo prefix.

bedroge commented 2 years ago

I'm not the real expert on Gentoo Prefix either (@amadio is one of the Prefix developers and often helping us out, so maybe he can correct me if I'm wrong), but Prefix also uses a custom ld.so.conf to make sure that stuff is picked up from the alternative sysroot:

$ cat /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/etc/ld.so.conf
# ld.so.conf autogenerated by env-update; make all changes to
# contents of /etc/env.d directory
include ld.so.conf.d/*.conf
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/local/lib64
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/local/lib

# and for libstdc++:
$ cat /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/etc/ld.so.conf.d/05gcc-x86_64-pc-linux-gnu.conf 
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0
/cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/10.3.0
haampie commented 2 years ago

Right, so I think I could add Gentoo Prefix support through a flag libtree --sys-root /path/to/gentoo/prefix [args] and then prepend that to the standard paths <sys-root>/lib, <sys-root>/etc/ld.so.conf, ...

Although there could be edge cases where elf files have an absolute path to a lib in their DT_NEEDED, in which case it probably should not prepend the prefix :thinking:

Edit: nope, seems like --sys-root would not work, since there are absolute paths including the prefix inside of ld.so.conf already, so it's quite different from chroot-type of environments.

haampie commented 2 years ago

On master you can now do this:

libtree --ldconf "$YOUR_PREFIX/etc/ld.so.conf" [elf files...]
amadio commented 2 years ago

The command lddtree is from app-misc/pax-utils from Gentoo. I saw libtree posted recently on HN, what a coincidence. In Gentoo Prefix standalone (which EESSI uses), the ELF interpreter is set to something inside the prefix, and using that interpreter to find libraries is important to find the correct answers (using /usr/bin/ldd will also show bad paths for prefix binaries/libs), as will lddtree from prefix, unless --prefix option is used. Only ldd from prefix shows correct answers by default for me:

$ lddtree $(which bash)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/bin/bash (interpreter => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2)
    libreadline.so.8 => /lib64/libreadline.so.8
        libtinfow.so.6 => /lib64/libtinfow.so.6
    libtinfo.so.6 => /lib64/libtinfo.so.6
    libc.so.6 => /lib64/libc.so.6
$ lddtree --prefix /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64 $(which bash)
/cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/bin/bash (interpreter => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2)
    libreadline.so.8 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libreadline.so.8
        libtinfow.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfow.so.6
    libtinfo.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfo.so.6
    libc.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/libc.so.6
$ ldd $(which bash)
    linux-vdso.so.1 (0x00007ffd8ebd1000)
    libreadline.so.8 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libreadline.so.8 (0x00007ff06a317000)
    libtinfo.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfo.so.6 (0x00007ff06a2d7000)
    libc.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/libc.so.6 (0x00007ff06a11e000)
    libtinfow.so.6 => /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/usr/lib64/libtinfow.so.6 (0x00007ff06a0de000)
    /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2 (0x00007ff06a3b7000)
$ /usr/bin/ldd $(which bash)
    linux-vdso.so.1 (0x00007ffc722e5000)
    libreadline.so.8 => /lib64/libreadline.so.8 (0x00007ff5878f1000)
    libtinfo.so.6 => /lib64/libtinfo.so.6 (0x00007ff5878b2000)
    libc.so.6 => /lib64/libc.so.6 (0x00007ff5876ad000)
    libtinfow.so.6 => /lib64/libtinfow.so.6 (0x00007ff58766c000)
    /cvmfs/sft.cern.ch/lcg/contrib/gentoo/linux/x86_64/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007ff587a5f000)

I'm not sure how you can support prefix with libtree, but looking at the implementation of lddtree (which is a shell or python script) may provide some good insights based on its support for the --prefix option.

haampie commented 2 years ago

Thanks, I believe /usr/bin/ldd used to show the correct output by simply executing the file with some variable set that hints to the dynamic linker that it should not actually execute. But this was considered a security problem when elf files had a different interpreter. So now ldd is probably just a wrapper that runs ld.so <executable> explicitly, but then it picks the wrong interpreter :D.

So, my current solution is to simply allow the user to set the path to ld.so.conf, assuming this contains all the paths that would otherwise be default search paths. Let me know if libtree --ldconf "$YOUR_PREFIX/etc/ld.so.conf" [elf files...] works for you.

haampie commented 2 years ago

The flag is now part of https://github.com/haampie/libtree/releases/tag/v3.0.0-rc8

ocaisa commented 2 years ago

This is looking good

[ocaisa@gnode1 release]$ libtree -vp --ldconf "$EPREFIX/etc/ld.so.conf" deviceQuery
deviceQuery 
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libstdc++.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/usr/lib/gcc/x86_64-pc-linux-gnu/11.1.0/libgcc_s.so.1 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/librt.so.1 [ld.so.conf]
│   ├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 [ld.so.conf]
│   └── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libc.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libm.so.6 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/ld-linux-x86-64.so.2 [ld.so.conf]
├── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libdl.so.2 [ld.so.conf]
└── /cvmfs/pilot.eessi-hpc.org/2021.06/compat/linux/x86_64/lib64/libpthread.so.0 [ld.so.conf]
haampie commented 2 years ago

Great, then I'll release version 3.0.0 soon :tm: