LLVM12: Building LLVM+clang for llvmtools fails

[dslm4515]

is this issue related to build llvm in llvmtools?

For LLVM 12, I just fine while building. See https://github.com/dslm4515/CMLFS/issues/12#issuecomment-869230916

Now moving to chroot steps. I'll report if an issue appeared.

Originally posted by @owl4ce in https://github.com/dslm4515/CMLFS/issues/14#issuecomment-869751706

So I switched back to building CMLFS with LLVM12. Still not successful. I accidentally used ninja instead of make:

 [1619/3634] Building CXX object projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/sanitizer_linux.cpp.o
FAILED: projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/sanitizer_linux.cpp.o 
/cgnutools/bin/x86_64-cmlfs-linux-musl-clang++  -DHAVE_RPC_XDR_H=0 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Iprojects/compiler-rt/lib/sanitizer_common -I../projects/compiler-rt/lib/sanitizer_common -Iinclude -I../include -I../projects/compiler-rt/lib/sanitizer_common/.. -g -g1  -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -fdiagnostics-color -ffunction-sections -fdata-sections -Wall -std=c++14 -Wno-unused-parameter -O3 -DNDEBUG    -m64 -fPIC -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -O3 -gline-tables-only -Wno-gnu -Wno-variadic-macros -Wno-c99-extensions -nostdinc++ -fno-rtti -Wframe-larger-than=570 -Wglobal-constructors -std=c++14 -MD -MT projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/sanitizer_linux.cpp.o -MF projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/sanitizer_linux.cpp.o.d -o projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/sanitizer_linux.cpp.o -c ../projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp
../projects/compiler-rt/lib/sanitizer_common/sanitizer_linux.cpp:30:10: fatal error: 'asm/param.h' file not found
#include <asm/param.h>
         ^~~~~~~~~~~~~
1 error generated.
[1620/3634] Building CXX object utils/yaml-bench/CMakeFiles/yaml-bench.dir/YAMLBench.cpp.o
ninja: build stopped: subcommand failed.

[dslm4515]

I used this command to configure the source:

CC=x86_64-cmlfs-linux-musl-clang \
CXX=x86_64-cmlfs-linux-musl-clang++ \
cmake -B build -G Ninja       \
      -DCMAKE_BUILD_TYPE=Release \
      -DCMAKE_INSTALL_PREFIX=/llvmtools \
      -DLLVM_LINK_LLVM_DYLIB=ON \
      -DLLVM_BUILD_LLVM_DYLIB=ON \
      -DLLVM_BUILD_TESTS=OFF \
      -DLLVM_ENABLE_LIBEDIT=OFF  \
      -DLLVM_ENABLE_LIBXML2=OFF \
      -DLLVM_ENABLE_LIBCXX=ON \
      -DLLVM_INCLUDE_GO_TESTS=OFF  \
      -DLLVM_INCLUDE_TESTS=OFF \
      -DLLVM_INCLUDE_DOCS=OFF \
      -DLLVM_INCLUDE_EXAMPLES=OFF \
      -DLLVM_INCLUDE_BENCHMARKS=OFF \
      -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-pc-linux-musl  \
      -DLLVM_HOST_TRIPLE="x86_64-pc-linux-musl" \
      -DLLVM_TARGET_ARCH="X86" \
      -DLLVM_TARGETS_TO_BUILD="X86"  \
      -DCOMPILER_RT_DEFAULT_TARGET_TRIPLE="x86_64-pc-linux-musl" \
      -DCOMPILER_RT_BUILD_SANITIZERS=OFF \
      -DCOMPILER_RT_BUILD_XRAY=OFF \
      -DCOMPILER_RT_BUILD_PROFILE=OFF \
      -DCOMPILER_RT_BUILD_LIBFUZZER=OFF \
      -DCOMPILER_RT_USE_BUILTINS_LIBRARY=ON \
      -DCLANG_DEFAULT_CXX_STDLIB=libc++ \
      -DCLANG_DEFAULT_UNWINDLIB=libunwind \
      -DCLANG_DEFAULT_RTLIB=compiler-rt \
      -DCLANG_DEFAULT_LINKER="/llvmtools/bin/ld.lld" \
      -DDEFAULT_SYSROOT="/llvmtools" \
      -DLLVM_ENABLE_LLD=ON \
      -DLLVM_ENABLE_RTTI=ON \
      -DLLVM_ENABLE_ZLIB=OFF \
      -DBacktrace_INCLUDE_DIR="/llvmtools/include" \
      -DBacktrace_LIBRARY="/llvmtools/lib/libexecinfo.so" \
      -DCMAKE_CXX_COMPILER_AR="/cgnutools/bin/llvm-ar" \
      -DCMAKE_C_COMPILER_AR="/cgnutools/bin/llvm-ar" \
      -DCMAKE_CXX_COMPILER_RANLIB="/cgnutools/bin/llvm-ranlib" \
      -DCMAKE_C_COMPILER_RANLIB="/cgnutools/bin/llvm-ranlib" \
      -DCMAKE_INSTALL_OLDINCLUDEDIR="/llvmtools/include" \
      -DCMAKE_LINKER="/cgnutools/bin/ld.lld" \
      -DCMAKE_NM="/cgnutools/bin/llvm-nm" \
      -DCMAKE_OBJCOPY="/cgnutools/bin/llvm-objcopy" \
      -DCMAKE_READELF="/cgnutools/bin/llvm-readelf" \
      -DCMAKE_STRIP="/cgnutools/bin/llvm-strip" \
      -DICONV_LIBRARY_PATH="/llvmtools/lib/libc.so"

[owl4ce]

My complete preference. I wrap every command with time for every build and install 😄

popd

# Rename the LLVM source directory to ${LLVMSRC}.
mv -v llvm-12.0.0.src "$LLVMSRC" && pushd "$LLVMSRC"

# Decompress clang, lld, and compiler-rt to correct directories.
pushd "${LLVMSRC}/projects/" && \
    tar xf ../../pkgs/compiler-rt-12.0.0.src.tar.xz && mv -v compiler-rt-12.0.0.src compiler-rt
popd

pushd "${LLVMSRC}/tools/" && \
    tar xf ../../pkgs/clang-12.0.0.src.tar.xz && mv -v clang-12.0.0.src clang
    tar xf ../../pkgs/lld-12.0.0.src.tar.xz   && mv -v lld-12.0.0.src lld
popd

# Apply patches (from Void Linux).
../patches/llvm-12/stage1-appatch

# Disable sanitizers for musl, fixing "early build failure".
sed -i 's|set(COMPILER_RT_HAS_SANITIZER_COMMON TRUE)|set(COMPILER_RT_HAS_SANITIZER_COMMON FALSE)|' \
projects/compiler-rt/cmake/config-ix.cmake

# Set default compiler to new symlink from Stage-0 Clang/LLVM.
# Sets C and C++ compiler's build flags to reduce debug symbols.
CC="${CMLFS_TARGET}-clang" CFLAGS="-g -g1"
CXX="${CMLFS_TARGET}-clang++" CXXFLAGS="-g -g1"
export CC CXX CFLAGS CXXFLAGS

# Update host/target triple detection.
cp -v ../files/config.guess cmake/

# Configure source.
cmake -B build \
    -DCMAKE_BUILD_TYPE=Release                                  \
    -DCMAKE_INSTALL_PREFIX="/llvmtools"                         \
    -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON                            \
    -DLLVM_LINK_LLVM_DYLIB=ON                                   \
    -DLLVM_BUILD_LLVM_DYLIB=ON                                  \
    -DLLVM_BUILD_TESTS=OFF                                      \
    -DLLVM_ENABLE_LIBEDIT=OFF                                   \
    -DLLVM_ENABLE_LIBXML2=OFF                                   \
    -DLLVM_ENABLE_LIBCXX=ON                                     \
    -DLLVM_INCLUDE_GO_TESTS=OFF                                 \
    -DLLVM_INCLUDE_TESTS=OFF                                    \
    -DLLVM_INCLUDE_DOCS=OFF                                     \
    -DLLVM_INCLUDE_EXAMPLES=OFF                                 \
    -DLLVM_INCLUDE_BENCHMARKS=OFF                               \
    -DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-pc-linux-musl"         \
    -DLLVM_HOST_TRIPLE="x86_64-pc-linux-musl"                   \
    -DLLVM_TARGET_ARCH="X86"                                    \
    -DLLVM_TARGETS_TO_BUILD="X86"                               \
    -DCOMPILER_RT_DEFAULT_TARGET_TRIPLE="x86_64-pc-linux-musl"  \
    -DCOMPILER_RT_BUILD_SANITIZERS=OFF                          \
    -DCOMPILER_RT_BUILD_XRAY=OFF                                \
    -DCOMPILER_RT_BUILD_PROFILE=OFF                             \
    -DCOMPILER_RT_BUILD_LIBFUZZER=OFF                           \
    -DCOMPILER_RT_USE_BUILTINS_LIBRARY=ON                       \
    -DCLANG_DEFAULT_CXX_STDLIB=libc++                           \
    -DCLANG_DEFAULT_UNWINDLIB=libunwind                         \
    -DCLANG_DEFAULT_RTLIB=compiler-rt                           \
    -DCLANG_DEFAULT_LINKER="/llvmtools/bin/ld.lld"              \
    -DDEFAULT_SYSROOT="/llvmtools"                              \
    -DLLVM_ENABLE_LLD=ON                                        \
    -DLLVM_ENABLE_RTTI=ON                                       \
    -DLLVM_ENABLE_ZLIB=OFF                                      \
    -DBacktrace_INCLUDE_DIR="/llvmtools/include"                \
    -DBacktrace_LIBRARY="/llvmtools/lib/libexecinfo.so"         \
    -DCMAKE_CXX_COMPILER_AR="/cgnutools/bin/llvm-ar"            \
    -DCMAKE_C_COMPILER_AR="/cgnutools/bin/llvm-ar"              \
    -DCMAKE_CXX_COMPILER_RANLIB="/cgnutools/bin/llvm-ranlib"    \
    -DCMAKE_C_COMPILER_RANLIB="/cgnutools/bin/llvm-ranlib"      \
    -DCMAKE_INSTALL_OLDINCLUDEDIR="/llvmtools/include"          \
    -DCMAKE_LINKER="/cgnutools/bin/ld.lld"                      \
    -DCMAKE_NM="/cgnutools/bin/llvm-nm"                         \
    -DCMAKE_OBJCOPY="/cgnutools/bin/llvm-objcopy"               \
    -DCMAKE_READELF="/cgnutools/bin/llvm-readelf"               \
    -DCMAKE_STRIP="/cgnutools/bin/llvm-strip"                   \
    -DICONV_LIBRARY_PATH="/llvmtools/lib/libc.so"

# Build.
time { make -C build; }

# Install, but some binaries are not installed, make sure to install important binaries.
time {
    pushd build/ && \
        cmake -DCMAKE_INSTALL_PREFIX="/llvmtools" -P cmake_install.cmake && \
        cp -v bin/llvm-as /llvmtools/bin/                                && \
        cp -v bin/llvm-readobj /llvmtools/bin/                           && \
        ln -sv llvm-readobj /llvmtools/bin/llvm-readelf                  && \
    popd && rm -rf build
}

# Since Binutils won't be used, create a symlink to LLVM tools and set lld as default toolchain linker.
for B in as ar ranlib readelf nm objcopy objdump size strip; do
    ln -sv llvm-${B} /llvmtools/bin/${B}
done
ln -sv lld /llvmtools/bin/ld

# Configure Stage-1 Clang to build binaries with "/llvmtools/lib/ld-musl-x86_64.so.1" instead of "/lib/ld-musl-x86_64.so.1".
ln -sv clang-12 /llvmtools/bin/x86_64-pc-linux-musl-clang   && \
ln -sv clang-12 /llvmtools/bin/x86_64-pc-linux-musl-clang++ && \
cat > /llvmtools/bin/x86_64-pc-linux-musl.cfg << "EOF"
-Wl,-dynamic-linker /llvmtools/lib/ld-musl-x86_64.so.1
EOF

# Unset exported flags and set the new PATH since "/cgnutools" isn't used anymore.
unset B CFLAGS CXXFLAGS
export PATH="/llvmtools/bin:/llvmtools/usr/bin:/bin:/usr/bin"

# Configure new Stage-1 Clang/LLVM environment.
sed -i "s|PATH=.*|PATH=\"${PATH}\"|" ~/.bashrc
sed -i '/unset CFLAGS CXXFLAGS/d' ~/.bashrc
cat >> ~/.bashrc << "EOF"
# Compiler environment
CC="${TARGET_TRUPLE}-clang"
CXX="${TARGET_TRUPLE}-clang++"
AR="llvm-ar"
AS="llvm-as"
RANLIB="llvm-ranlib"
LD="ld.lld"
STRIP="llvm-strip"
export CC CXX AR AS RANLIB LD STRIP
EOF
source ~/.bashrc

popd

I created script like you for patch from Void Linux.

#!/usr/bin/env bash
# Stage-1 Clang/LLVM.

pushd "${LLVMSRC}/projects/" && \
    for P in \
        compiler-rt-aarch64-ucontext.patch \
        compiler-rt-sanitizer-ppc64-musl.patch \
        compiler-rt-size_t.patch \
        compiler-rt-xray-ppc64-musl.patch
    do patch -Np1 -i ../../patches/llvm-12/${P}
    done; unset P
popd

pushd "${LLVMSRC}/tools/" && \
    for P in \
        clang-001-fix-unwind-chain-inclusion.patch \
        clang-002-add-musl-triples.patch \
        clang-003-ppc64-dynamic-linker-path.patch \
        clang-004-ppc64-musl-elfv2.patch
    do patch -Np1 -i ../../patches/llvm-12/${P}
    done; unset P
popd

pushd "${LLVMSRC}/../" && \
    for P in \
        llvm-001-musl.patch \
        llvm-002-musl-ppc64-elfv2.patch \
        llvm-003-ppc-secureplt.patch \
        llvm-004-override-opt.patch \
        llvm-005-ppc-bigpic.patch \
        llvm-006-aarch64-mf_exec.patch
    do patch  -Np1 -i ./patches/llvm-12/${P}
    done; unset P
popd

[owl4ce]

Also I use the latest config.guess file. https://git.savannah.gnu.org/cgit/config.git/plain/config.guess

[owl4ce]

Have you already disabled the sanitizers?

# Disable sanitizers for musl, fixing "early build failure".
sed -i 's|set(COMPILER_RT_HAS_SANITIZER_COMMON TRUE)|set(COMPILER_RT_HAS_SANITIZER_COMMON FALSE)|' \
projects/compiler-rt/cmake/config-ix.cmake

[dslm4515]

Have you already disabled the sanitizers?

# Disable sanitizers for musl, fixing "early build failure".
sed -i 's|set(COMPILER_RT_HAS_SANITIZER_COMMON TRUE)|set(COMPILER_RT_HAS_SANITIZER_COMMON FALSE)|' \
projects/compiler-rt/cmake/config-ix.cmake

I thought I did... I’ll do it again!

[dslm4515]

@owl4ce , what software sorcery is this?? Lol. WOW. I used the cmake command you posted and LLVM built without error!!!

[owl4ce]

I'm going to rebuild llvmtools with purpose removes this line when configuring LLVM.

-DLLVM_INSTALL_TOOLCHAIN_ONLY=ON

[owl4ce]

I wonder, this is ambigous flags. May I ask, is this correct?

https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/3-chroot/026-llvm#L81 https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/3-chroot/026-llvm#L105 https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/3-chroot/026-llvm#L87-L91

[dslm4515]

https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/3-chroot/026-llvm#L81

No, I believe this does not set the ‘root’ for the llvmtools tools to resolve its libraries and binaries. This is the root that LLVM will install and resolve its libraries and binaries... I have yet to test to confirm. Should be set to /usr in chroot.

https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/3-chroot/026-llvm#L87-L91

This is set to force the build to use LLVM’s binary tools. This is important as I chose to install binutils and GCC in /usr Instead of /opt/gnu and in /llvmtools instead of /llvmtools/gnu

[owl4ce]

https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/3-chroot/026-llvm#L81

No, I believe this does not set the ‘root’ for the llvmtools tools to resolve its libraries and binaries. This is the root that LLVM will install and resolve its libraries and binaries... I have yet to test to confirm. Should be set to /usr in chroot.

So, you should remove one of two DEFAULT_SYSROOT. But I still don't know why double AR and LINKER flags? While in llvmtools is uses like below. https://github.com/dslm4515/CMLFS/blob/52fd160898cd382f789c532b3488b4b3d03437ae/2-llvmtools/06-clang-llvm#L90-L91

[dslm4515]

So, you should remove one of two DEFAULT_SYSROOT.

yes, you are right. I did not notice.

[dslm4515]

But I still don't know why double AR and LINKER flags? While in llvmtools is uses like below.

yeah, I need to check that and remove doubles

[dslm4515]

I do appreciate you catching those!

[dslm4515]

So, you should remove one of two DEFAULT_SYSROOT.

yes, you are right. I did not notice.

Yes. I just confirmed: Setting -DDEFAULT_SYSROOT="/llvmtols" will produce clang that will search for headers in /llvmtools/include and libraries in /llvmtools/lib.

Therefore -DDEFAULT_SYSROOT="/usr" is what sets the built LLVM to /usr to search for headers in /usr/include and libraries in /usr/lib