search

dslm4515 / CMLFS

Clang-Built Musl Linux From Scratch
MIT License
99 stars 18 forks source link

LLVM-17.0.5: Stage 2 Clang has sysroot set as /llvmtools #94

Closed dslm4515 closed 2 months ago

dslm4515 commented 7 months ago

After building & installing LLVM (stage2, aka final system's LLVM), clang was built with sysroot set to /llvmtools. This is likely inherited from stage1 LLVM installed in /llvmtools.

Proof:

$ echo "int main(){}" > dummy.c
$ /usr/bin/clang dummy.c -v -Wl,--verbose &> dummy.log

When viewing dummy.log, clang is executed these "built-in" arguments:

  "/usr/bin/clang-17" -cc1 -triple x86_64-pc-linux-musl -emit-obj -mrelax-all -dumpdir a- -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name dummy.c -mrelocation-model pic -pic-level 2 -pic-is-pie -mframe-pointer=all -ffp-contract=on -fno-rounding-math -mconstructor-aliases -funwind-tables=2 -target-cpu x86-64 -tune-cpu generic -debugger-tuning=gdb -v -fcoverage-compilation-dir=/sources -resource-dir /usr/lib/clang/17 -isysroot /llvmtools -internal-isystem /llvmtools/usr/local/include -internal-externc-isystem /llvmtools/usr/include/fortify -internal-externc-isystem /llvmtools/usr/include -internal-isystem /usr/lib/clang/17/include -fdebug-compilation-dir=/sources -ferror-limit 19 -stack-protector 2 -fgnuc-version=4.2.1 -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/dummy-d7adf8.o -x c dummy.c

And then clang looks for lld/ld in first in /llvmtools/bin instead of /usr/bin:

"/llvmtools/bin/ld.lld" --sysroot=/llvmtools  -pie -z now -z relro --hash-style=gnu --build-id --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib/ld-musl-x86_64.so.1 -o a.out /llvmtools/lib/Scrt1.o /llvmtools/lib/crti.o /usr/lib/clang/17/lib/x86_64-pc-linux-musl/clang_rt.crtbegin.o -L/usr/lib/clang/17/lib/x86_64-pc-linux-musl -L/llvmtools/lib/ -L/llvmtools/lib --as-needed /tmp/dummy-d7adf8.o --verbose /usr/lib/clang/17/lib/x86_64-pc-linux-musl/libclang_rt.builtins.a --as-needed -lunwind --no-as-needed --push-state --as-needed -latomic --pop-state -lc /usr/lib/clang/17/lib/x86_64-pc-linux-musl/libclang_rt.builtins.a --as-needed -lunwind --no-as-needed /usr/lib/clang/17/lib/x86_64-pc-linux-musl/clang_rt.crtend.o /llvmtools/lib/crtn.o

Which then results in these objects compiled together:

ld.lld: /llvmtools/lib/Scrt1.o
ld.lld: /llvmtools/lib/crti.o
ld.lld: /usr/lib/clang/17/lib/x86_64-pc-linux-musl/clang_rt.crtbegin.o
ld.lld: /tmp/dummy-d7adf8.o
ld.lld: /usr/lib/clang/17/lib/x86_64-pc-linux-musl/libclang_rt.builtins.a
ld.lld: /llvmtools/lib/libunwind.so
ld.lld: /llvmtools/lib/libatomic.so
ld.lld: /llvmtools/lib/libc.so
ld.lld: /usr/lib/clang/17/lib/x86_64-pc-linux-musl/libclang_rt.builtins.a
ld.lld: /llvmtools/lib/libunwind.so
ld.lld: /usr/lib/clang/17/lib/x86_64-pc-linux-musl/clang_rt.crtend.o
ld.lld: /llvmtools/lib/crtn.o

Even setting the sysroot with CMake, -DDEFAULT_SYSROOT="/usr", clang still builds with sysroot as /llvmtools. Perhaps source needs a patch to enforce this?

dslm4515 commented 7 months ago

A possible workaround is to add a clang configuration file:

$ cat > /usr/bin/${TUPLE}.cfg << "EOF"
--sysroot=/
EOF

But clang still looks for lld/ld in /llvmtools/bin

dslm4515 commented 7 months ago

Another workaround is rebuilding clang with the "tainted" clang.

Big disadvantage is build time. As of LLVM 17.0.5, there is no "standalone build" of clang. In other words, clang cannot be built against a system installed LLVM... it has to built with the monorepo LLVM-project source tree. Apparently, developing standalone builds add "complication to the build system" source: discourse.llvm.org.

When re-building clang, LLVM support libraries will also be built. For systems with a package-manager or packager, this is easy as:

  1. Rebuild clang with installed tainted clang via LLVM monorepo
  2. Package only the rebuilt clang without the rebuilt LLVM libraries & tools (as long as tainted clang & rebuilt clang were compiled from the same patched LLVM monorepo)
  3. Uninstall tainted clang
  4. Install rebuilt clang

For other systems, this may be tricky. Probably will need to do the following:

  1. Rebuild clang with installed tainted clang via LLVM monorepo
  2. Install rebuilt clang and LLVM libraries & tools to a directory, say DESTDIR=/BUILD
  3. Move clang tools, headers, and libraries from /BUILD to another directory, say /CLANG-BUILD (as long as tainted clang & rebuilt clang were compiled from the same patched LLVM monorepo)
  4. Copy /CLANG-BUILD to / ... this should overwrite the tainted clang
dslm4515 commented 7 months ago

Fixed issue by rebuilding clang, commit b6f0fef

dslm4515 commented 2 months ago

Cross-compiling for i686 on an AMD64 host.

Rebuilding stage2 clang results in clang (stage2,r2) built with the sysroot at /usr instead of / ... and searching for system headers in /usr/usr/include:

ignoring nonexistent directory "/usr/usr/local/include"
ignoring nonexistent directory "/usr/usr/include/fortify"
ignoring nonexistent directory "/usr/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/clang/17/include

When configuring the source, a new build directory was made and -DDEFAULT_SYSROOT=/ set.... but clang still has the sysroot set to /usr. The previous clang (stage2,r1) used a config file to set the sysroot & header search paths & order.

Perhaps, stage 2 r1 clang should not use a config when building stage 2 r2 clang?

dslm4515 commented 2 months ago

stage 2 r1 clang has this configuration when no config file present in /bin:

Will try to build stage 2 r2 clang without a clang config present (/bin/${TUPLE}.cfg)

dslm4515 commented 2 months ago

Nope. Build fails right away:

error: <cstddef> tried including <stddef.h> but didn't find libc++'s <stddef.h> header. 
          This usually means that your header search paths are not configured properly. 
          The header search paths should contain the C++ Standard Library headers before 
          any C Standard Library, and you are probably using compiler flags that make that 
          not be the case.
dslm4515 commented 2 months ago

I'll try the build again but modify the clang config in /bin to not set the sysroot to /. This means the sysroot will be llvmtools.

cat >> /bin/${TUPLE}.cfg << "EOF"
-nostdinc++
-L/usr/lib
-I/usr/include/c++/v1
-I/usr/include
EOF
dslm4515 commented 2 months ago

No change. Same error of rebuild clang having sysroot at /usr with header seaches set to /usr/usr/include

dslm4515 commented 2 months ago

Now, i did notice that stage 2,r0 clang has a sysroot set in a header:

$  grep SYSROOT /usr/include/clang/Config/config.h
#define DEFAULT_SYSROOT "/usr"

Is this where the sysroot is set when building rebuilding stage 2 clang? Does the -DDEFAULT_SYSROOT=/ CMake option only set it for the build's header (include/clang/Config/config.h)?

I did compare stage 2 clang's from a previous build (LLVM-17.0.5), and the first stage 2 clang had it's config header with #define DEFAULT_SYSROOT "/usr" ... so perhaps changing it will not change the sysroot for the rebuilt clang?

dslm4515 commented 2 months ago

YES! Changing the stage 2 clang's /usr/include/clang/Config/config.h does set the sysroot of the rebuilt clang!

dslm4515 commented 2 months ago

Technically, adjusting the header of stage 1 clang at /llvmtools//include/clang/Config/config.h right before building stage2 LLVM will negate the need to rebuild stage 2 clang

Refer to commit 39207aa057333afaf1845b04a09fd2f01266a61b