CMLFS/3-chroot/010-utmps: utmps Segmentation Fault

dvdfreitag commented 2 years ago

I have found the reason why utmps fails to build using Clang 12 from /llvmtools/bin. Since I am trying to build the end system without gcc or binutils, this was a pretty big issue for me.

After some short investigation, I found that somehow the static binaries generated by utmps were claiming to have an interpreter:

# file utmps-utmpd
utmps-utmpd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped

And when debugging with lldb:

* thread #1, name = 'utmps-utmpd', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x00007ffff7fbe950 libc.so`decode_vec(v=0x0000000000000000, a=0x00007fffffffe118, cnt=32) at dynlink.c:198:10

But ldd correctly recognizes the binaries:

$ ldd utmps-utmpd
ldd: utmps-utmpd: Not a valid dynamic program

I found it extremely strange why a statically linked binary would have an interpreter listed by file, and why lldb showed the source of the Segmentation Fault was from within libc.so. So I did a bit more digging.

With CC=x86_64-cmlfs-linux-musl-clang, Clang seems to always pass -dynamic-linker, even for -static (for brevity the linked objects are replaced with ...):

"/llvmtools/bin/ld.lld" --sysroot=/usr --eh-frame-hdr -m elf_x86_64 -static -o utmps-utmpd /usr/lib/crt1.o /usr/lib/crti.o /llvmtools/lib/clang/12.0.0/lib/linux/clang_rt.crtbegin-x86_64.o -L/usr/lib -dynamic-linker /lib/ld-musl-x86_64.so.1 ... -lskarnet --verbose --start-group /llvmtools/lib/clang/12.0.0/lib/linux/libclang_rt.builtins-x86_64.a -l:libunwind.a -lc --end-group /llvmtools/lib/clang/12.0.0/lib/linux/clang_rt.crtend-x86_64.o /usr/lib/crtn.o

With CC=clang, the -dynamic-linker flag is no longer present, however some of the paths are broken, and I must manually pass -L/usr/lib:

"/llvmtools/bin/ld.lld" --sysroot=/llvmtools --eh-frame-hdr -m elf_x86_64 -static -o utmps-utmpd /llvmtools/bin/../lib/crt1.o /llvmtools/bin/../lib/crti.o /llvmtools/lib/clang/12.0.0/lib/linux/clang_rt.crtbegin-x86_64.o -L/usr/lib -L/llvmtools/bin/../lib -L/llvmtools/lib ... -lskarnet --verbose --start-group /llvmtools/lib/clang/12.0.0/lib/linux/libclang_rt.builtins-x86_64.a -l:libunwind.a -lc --end-group /llvmtools/lib/clang/12.0.0/lib/linux/clang_rt.crtend-x86_64.o /llvmtools/bin/../lib/crtn.o

Here's how those are symlinked:

# ls -lah `which x86_64-cmlfs-linux-musl-clang`
lrwxrwxrwx 1 root root 8 Jun 29 07:23 /llvmtools/bin/x86_64-cmlfs-linux-musl-clang -> clang-12
# ls -lah `which clang`
lrwxrwxrwx 1 root root 8 Jun 29 03:44 /llvmtools/bin/clang -> clang-12

Moreover, when building with CC=clang, the build not only succeeds, utmps-utmpd runs without issue:

# ./utmps-utmpd 
utmps-utmpd: fatal: unable to get $IPCREMOTEEUID from environment

This seems to suggest to me that the out of tree cross compilation of LLVM for /llvmtools is somehow broken.

The host system is musl Void Linux running under WSL 2 on x86_64. The target system is also x86_64.

dvdfreitag commented 2 years ago

On further investigation, it seems this is the issue, from CMLFS/3-chroot/008-Adjust_Toolchain:

echo " --sysroot=/usr -Wl,-dynamic-linker /lib/ld-musl-$(uname -m).so.1" > \
     /llvmtools/bin/$(uname -m)-cmlfs-linux-musl.cfg

Here we are manually instructing clang to always insert -dynamic-linker, but clang can already build working dynamic libraries with the correct Interpreter:

# echo "int main(){}" > t.c
# clang -o t t.c --verbose -Wl,--verbose
"/llvmtools/bin/clang-12" -cc1 -triple x86_64-pc-linux-musl -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name t.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu x86-64 -tune-cpu generic -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /llvmtools/lib/clang/12.0.0 -isysroot /llvmtools -internal-isystem /llvmtools/usr/local/include -internal-externc-isystem /llvmtools/include -internal-externc-isystem /llvmtools/usr/include -internal-isystem /llvmtools/lib/clang/12.0.0/include -fdebug-compilation-dir /sources/src/utmps-system/temp -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/t-edcaf0.o -x c t.c
# file t
t: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped

If I remove the -dynamic-linker section from the config file,

echo " --sysroot=/usr" > /llvmtools/bin/$(uname -m)-cmlfs-linux-musl.cfg

And rebuild:

# x86_64-cmlfs-linux-musl-clang -o t t.c --verbose -Wl,--verbose
Configuration file: /llvmtools/bin/x86_64-cmlfs-linux-musl.cfg
"/llvmtools/bin/clang-12" -cc1 -triple x86_64-cmlfs-linux-musl -emit-obj -mrelax-all --mrelax-relocations -disable-free -disable-llvm-verifier -discard-value-names -main-file-name t.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu x86-64 -tune-cpu generic -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /llvmtools/lib/clang/12.0.0 -isysroot /usr -internal-isystem /usr/usr/local/include -internal-externc-isystem /usr/include -internal-externc-isystem /usr/usr/include -internal-isystem /llvmtools/lib/clang/12.0.0/include -fdebug-compilation-dir /sources/src/utmps-system/temp -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -faddrsig -o /tmp/t-9e1d1a.o -x c t.c
# file t
t: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-x86_64.so.1, not stripped

Once this change has been made, with CC=x86_64-cmlfs-linux-musl-clang:

# file utmps-utmpd 
utmps-utmpd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, not stripped
# ./utmps-utmpd 
utmps-utmpd: fatal: unable to get $IPCREMOTEEUID from environment

utmps succeeds to build, and works without Segmentation Fault

dslm4515 commented 2 years ago

I created CMLFS/3-chroot/008-Adjust_Toolchain because the llvmtools toolchain produced binaries that looked for an interpreter in /llvmtools instead on the final system root.

LLVM docs don't seem to be much help when trying to build sandboxed toolchains.