fhahn
commented
1 year ago I assume it may be better to provide you with a reproducer?
Yes, unfortunately without a reproducer there's not much we can do to fix that, especially the build above uses PGO.
Closed vient closed 7 months ago
fhahn
commented
1 year ago I assume it may be better to provide you with a reproducer?
Yes, unfortunately without a reproducer there's not much we can do to fix that, especially the build above uses PGO.
vient
commented
1 year ago Here is the reproducer that worked on my system. Only LLVM code is now used: I compile clang 17.0.0-rc3, compile profiled version, gather profiles on clang itself and then try to use that profile.
Generated clang.profdata which is used in the "build PGO compiler" step
Edit: updated reproducer to work in ubuntu:23.04 docker container.
vient
commented
11 months ago Removed unneeded parts from reproducer, confirmed that it reproduces on 17.0.1 Reproduces only with LTO=Full but not with Thin or Off.
Save this to repro_64929.sh and run docker run -it -v $PWD:/test ubuntu:23.04 /test/repro_64929.sh
#!/bin/bash -eux
apt update
apt install --yes git apt-transport-https build-essential cmake coreutils curl libc6-dev-i386 linux-tools-generic psmisc python3 python3-pip sed software-properties-common vim wget
python3 -m pip install --break-system-packages ninja
LLVM_GIT_BRANCH="llvmorg-17.0.1"
TOPLEV="$(dirname "$(realpath -s "${BASH_SOURCE[0]}")")"
LLVM_PROJECT_DIR="${TOPLEV}/llvm-project"
BASE_COMPILER_DIR="${TOPLEV}/stage1"
BINUTILS_INCLUDE_DIR="${TOPLEV}/binutils/include"
mkdir -p "${TOPLEV}"
cd "$_"
git clone --depth 1 git://sourceware.org/git/binutils-gdb.git binutils
git clone -b "${LLVM_GIT_BRANCH}" --depth=1 https://github.com/llvm/llvm-project.git
# build base compiler
mkdir -p "${BASE_COMPILER_DIR}"
cd "$_"
cmake -G Ninja "${LLVM_PROJECT_DIR}/llvm" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX="${BASE_COMPILER_DIR}/install" \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;lld" \
-DLLVM_ENABLE_RUNTIMES="compiler-rt" \
-DLLVM_TARGETS_TO_BUILD=X86
ninja install
ninja -t clean
# build self-profiling compiler
mkdir -p "${TOPLEV}/stage2-prof-gen"
cd "$_"
CPATH="${BASE_COMPILER_DIR}/install/bin"
LLVM_EXE_LINKER_FLAGS="-fuse-ld=lld"
LLVM_SHARED_LINKER_FLAGS="-fuse-ld=lld"
cmake -G Ninja "${LLVM_PROJECT_DIR}/llvm" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER="${CPATH}/clang" \
-DCMAKE_C_FLAGS="-Xclang -mllvm -Xclang -vp-counters-per-site=256" \
-DCMAKE_CXX_COMPILER="${CPATH}/clang++" \
-DCMAKE_CXX_FLAGS="-Xclang -mllvm -Xclang -vp-counters-per-site=256" \
-DCMAKE_EXE_LINKER_FLAGS="${LLVM_EXE_LINKER_FLAGS}" \
-DCMAKE_SHARED_LINKER_FLAGS="${LLVM_SHARED_LINKER_FLAGS}" \
-DCMAKE_INSTALL_PREFIX="${TOPLEV}/stage2-prof-gen/install" \
-DLLVM_BINUTILS_INCDIR="${BINUTILS_INCLUDE_DIR}" \
-DLLVM_BUILD_INSTRUMENTED=IR \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_PROFILE_DATA_DIR="${TOPLEV}/stage2-prof-gen/profiles/%p" \
-DLLVM_TARGETS_TO_BUILD=X86 \
-DLLVM_USE_LINKER=lld
ninja install
ninja -t clean
# gather compiler profiles
CPATH="${TOPLEV}/stage2-prof-gen/install/bin"
mkdir -p "${TOPLEV}/stage3-train-llvm-release"
cd "$_"
cmake -G Ninja "${LLVM_PROJECT_DIR}/llvm" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER="${CPATH}/clang" \
-DCMAKE_CXX_COMPILER="${CPATH}/clang++" \
-DCMAKE_INSTALL_PREFIX="${PWD}/install" \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra" \
-DLLVM_TARGETS_TO_BUILD=X86 \
-DLLVM_USE_LINKER=lld
ninja bin/llvm-addr2line # random small target
rm -rf "${TOPLEV}/stage3-train"*
# merge profiles
cd "${TOPLEV}/stage2-prof-gen/profiles"
"${BASE_COMPILER_DIR}/install/bin/llvm-profdata" merge -output=clang.profdata ./*
# use PGO
mkdir -p "${TOPLEV}/stage2-prof-use-lto"
cd "$_"
CPATH="${BASE_COMPILER_DIR}/install/bin"
cmake -G Ninja "${LLVM_PROJECT_DIR}/llvm" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER="${CPATH}/clang" \
-DCMAKE_CXX_COMPILER="${CPATH}/clang++" \
-DCMAKE_INSTALL_PREFIX="${TOPLEV}/stage2-prof-use-lto/install" \
-DLLVM_BINUTILS_INCDIR="${BINUTILS_INCLUDE_DIR}" \
-DLLVM_ENABLE_LTO=Full \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_PROFDATA_FILE="${TOPLEV}/stage2-prof-gen/profiles/clang.profdata" \
-DLLVM_TARGETS_TO_BUILD=X86 \
-DLLVM_USE_LINKER=lld
ninja bin/dsymutil # fails here@vient Thanks for checking with the latest released version and for closing!
Compiling LLVM 17.0.0-rc3 with full LTO and PGO, got the following error
Link commands look like this
I assume it may be better to provide you with a reproducer? I don't have it at the moment since it occurred in CI.