unittests build does not respect CMAKE_EXE_LINKER_FLAGS, PipSqueak fails to link for MemorySanitizer

[FlashSheridan]

Summary

Trying to build our LLVM fork for MemorySanitizer with a local libc++ per MemorySanitizerBootstrappingClang fails with “ld.lld: error: unable to find library -lc++” trying to link unittests/Support/DynamicLibrary/CMakeFiles/SecondLib.dir/PipSqueak.cpp.o. If I add add_link_options(-L<my custom libc++ path>) to llvm/unittests/CMakeLists.txt, the file links. Except for the unittests, this directory is correctly propagated to the Clang++ command from the -DCMAKE_EXE_LINKER_FLAGS argument, as in MemorySanitizerBootstrappingClang#building-clang.

Details

(See also “Configuration Workarounds” below.) If I add the following arguments to our build system (a front-end to CMake and Ninja)

--debug-trycompile -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS="clang;lld" -DLLVM_USE_SANITIZER=MemoryWithOrigins -DLLVM_ENABLE_LIBCXX=ON -DCMAKE_CXX_FLAGS="-nostdinc++ -isystem $LIBCXX_DIR/include -isystem $LIBCXX_DIR/include/c++/v1" -DCMAKE_EXE_LINKER_FLAGS="-stdlib=libc++ -Wl,--rpath=$LIBCXX_DIR/lib/x86_64-unknown-linux-gnu -L$LIBCXX_DIR/lib/x86_64-unknown-linux-gnu -fsanitize=memory"

the Clang++ command for the failing link step contains my custom include directories, but is missing the lib directory, so it is understandably “unable to find library -lc++”: (Expanded with (\s"?-+) ⇒ \n\1 for clarity)

: && /usr/bin/clang++
 -fPIC
 -nostdinc++
 -isystem /home/zekebot/Documents/Code/External_Code/Llvm-15.0.4_for_libcxx/build_libcxx//include
 -isystem /home/zekebot/Documents/Code/External_Code/Llvm-15.0.4_for_libcxx/build_libcxx//include/c++/v1
 -stdlib=libc++
 -fvisibility-inlines-hidden
 -Wall
 -Wextra
 -Wno-unused-parameter
 -Wwrite-strings
 -Wcast-qual
 -Wmissing-field-initializers
 -pedantic
 -Wno-long-long
 -Wno-noexcept-type
 -Wnon-virtual-dtor
 -Wdelete-non-virtual-dtor
 -Wsuggest-override
 -Wno-comment
 -fsanitize=memory
 -fsanitize-memory-track-origins
 -fdiagnostics-color
 -O3
 -DNDEBUG 
 -stdlib=libc++ 
 -fuse-ld=lld
 -Wl,--color-diagnostics  
 -fsanitize=memory
 -fsanitize-recover=memory
 -shared 
 -o unittests/Support/DynamicLibrary/PipSqueak.so unittests/Support/DynamicLibrary/CMakeFiles/PipSqueak.dir/PipSqueak.cpp.o   && :

Configuration

Ubuntu 22.04.1 LTS (Jammy Jellyfish) x86_64 Ubuntu Clang version 15.0.6 Target: x86_64-pc-linux-gnu

Configuration Workarounds

• $LIBCXX_DIR/lib in MemorySanitizerBootstrappingClang doc needs platform name appended  (https://github.com/google/sanitizers/issues/1661)
• https://github.com/llvm/llvm-project/issues/57104#issuecomment-1278598494

See Also

• MemorySanitizerBootstrappingClang Wiki building instructions out of date #1230
• MemorySanitizerLibcxxHowTo wiki page has deprecated build instructions. #1574

[FlashSheridan]

Later in the breakage workaround process (which has been quite extensive, I get the impression that msan is not maintained — corrections welcome), I needed a similar kludge for llvm/lib/Transforms/Hello/CMakeLists.txt.

[FlashSheridan]

I eventually added the add_link_options kludge for llvm/CMakeLists.txt, llvm/lib/CMakeLists.txt, llvm/lib/LTO/CMakeLists.txt, llvm/lib/Transforms/Hello/CMakeLists.txt, and llvm/unittests/CMakeLists.txt, but it didn’t suffice. I got unexplained build failures with llvm-tblgen, which I could work around by substituting a non-sanitized version via LLVM_TABLEGEN. I then got unexplained build failures with clang-tblgen, which doesn’t seem to have a documented workaround. I got somewhat farther by building my own and symlinking it into the build tree, but it eventually got overwritten and I got more unexplained failures, e.g. at “[107/1485] Building Attrs.inc.” Given that 107 is less than 1485, this seemed unlikely to be a good use of time.