LLVM 18.1: C++ exceptions are broken in the MSan build

[ns-osmolsky]

It looks like C++ exceptions are broken when using MSan with libcxx. Here is the minimum test program:

#include <iostream>

int main(int argc, char **argv)
{
    try {
        std::cout << "Testing C++ exception...\n";
        throw std::runtime_error("boo");
    } catch(std::exception &ex) {
        std::cout << "Exception worked! (" << ex.what() << ")\n";
    }

    return 0;
}

Output:

Testing C++ exception...
MemorySanitizer:DEADLYSIGNAL
==3740922==ERROR: MemorySanitizer: stack-overflow on address 0x7ffe51dc5c38 (pc 0x7fde6dc661fc bp 0x7ffe51dc60a0 sp 0x7ffe51dc5c40 T3740922)
MemorySanitizer:DEADLYSIGNAL
MemorySanitizer: nested bug in the same thread, aborting.

Linkage:

$ ldd test | grep llvm
        libc++.so.1 => /opt/llvm-18/lib-msan/libc++.so.1 (0x00007fee9257d000)
        libc++abi.so.1 => /opt/llvm-18/lib-msan/libc++abi.so.1 (0x00007fee922c8000)
        libunwind.so.1 => /opt/llvm-18/lib-msan/libunwind.so.1 (0x00007fee920ab000)

[ns-osmolsky]

It looks like the process runs out of stack space:

...
...
#17256 0x00007ffff76a986f in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister(bool) ()
   from /opt/llvm-18/lib-msan/libunwind.so.1
#17257 0x00007ffff76a4e64 in unw_init_local () from /opt/llvm-18/lib-msan/libunwind.so.1
#17258 0x00007ffff76b929d in _Unwind_Backtrace () from /opt/llvm-18/lib-msan/libunwind.so.1
#17259 0x00000000004bd0ac in __sanitizer::BufferedStackTrace::UnwindSlow(unsigned long, unsigned int) ()
#17260 0x00000000004b591a in __sanitizer::BufferedStackTrace::Unwind(unsigned int, unsigned long, unsigned long, void*, unsigned long, unsigned long, bool)
    ()
#17261 0x0000000000430da0 in __sanitizer::BufferedStackTrace::UnwindImpl(unsigned long, unsigned long, void*, bool, unsigned int) ()
#17262 0x0000000000430ece in __msan::PrintWarningWithOrigin(unsigned long, unsigned long, unsigned int) [clone .part.0] ()
#17263 0x000000000043165e in __msan_warning_with_origin_noreturn ()
#17264 0x00007ffff76a9175 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getReg(int) ()
   from /opt/llvm-18/lib-msan/libunwind.so.1
#17265 0x00007ffff76a986f in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister(bool) ()
   from /opt/llvm-18/lib-msan/libunwind.so.1
#17266 0x00007ffff76a4e64 in unw_init_local () from /opt/llvm-18/lib-msan/libunwind.so.1
#17267 0x00007ffff76b929d in _Unwind_Backtrace () from /opt/llvm-18/lib-msan/libunwind.so.1
#17268 0x00000000004bd0ac in __sanitizer::BufferedStackTrace::UnwindSlow(unsigned long, unsigned int) ()
#17269 0x00000000004b591a in __sanitizer::BufferedStackTrace::Unwind(unsigned int, unsigned long, unsigned long, void*, unsigned long, unsigned long, bool)
    ()
#17270 0x0000000000430da0 in __sanitizer::BufferedStackTrace::UnwindImpl(unsigned long, unsigned long, void*, bool, unsigned int) ()
#17271 0x0000000000430ece in __msan::PrintWarningWithOrigin(unsigned long, unsigned long, unsigned int) [clone .part.0] ()
#17272 0x000000000043165e in __msan_warning_with_origin_noreturn ()
#17273 0x00007ffff76a9175 in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::getReg(int) ()
   from /opt/llvm-18/lib-msan/libunwind.so.1
#17274 0x00007ffff76a986f in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister(bool) ()
   from /opt/llvm-18/lib-msan/libunwind.so.1
#17275 0x00007ffff76a4e64 in unw_init_local () from /opt/llvm-18/lib-msan/libunwind.so.1
#17276 0x00007ffff76b5a4b in _Unwind_RaiseException () from /opt/llvm-18/lib-msan/libunwind.so.1
#17277 0x00007ffff795ee30 in __cxa_throw () from /opt/llvm-18/lib-msan/libc++abi.so.1
#17278 0x00000000004c61f4 in main ()

[ns-osmolsky]

Folks, any hints as to what is wrong here?

BTW, here is how I built the instrumented libcxx:

cmake \
        -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \
        -S runtimes -B llvm-build-msan \
        -DLLVM_USE_SANITIZER=MemoryWithOrigins \
        -DCMAKE_BUILD_TYPE=Release \
        -DCMAKE_INSTALL_PREFIX:PATH=$wd/llvm-install \
        -DLLVM_TARGETS_TO_BUILD="X86" \
        -DCMAKE_C_COMPILER=$destination/bin/clang \
        -DCMAKE_CXX_COMPILER=$destination/bin/clang++
make -C llvm-build-msan -j $CPUS cxx cxxabi unwind
make -C llvm-build-msan -j $CPUS install-cxx install-cxxabi install-unwind

[ns-osmolsky]

@ldionne could you please take a look at this? What am I missing from the libcxx/msan build recipe?

[ns-osmolsky]

For the record, C++ exceptions do work correctly when I build libcxx normally (i.e. without msan).

[ns-osmolsky]

Found a (silly) workaround for the msan-instrumented build of libcxx:

-DLIBCXXABI_USE_LLVM_UNWINDER=FALSE \

[ldionne]

Is it possible that libunwind is not being built with MemoryWithOrigins even though -DLLVM_USE_SANITIZER=MemoryWithOrigins is passed? Can you check whether libunwind is being built with msan support?

[ns-osmolsky]

Is it possible that libunwind is not being built with MemoryWithOrigins even though -DLLVM_USE_SANITIZER=MemoryWithOrigins is passed? Can you check whether libunwind is being built with msan support?

That guess appears to be right on the money!

$ ldd llvm-install/lib/libunwind.so
        linux-vdso.so.1 (0x00007ffc095e4000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f1ebe4e4000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1ebe4c1000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1ebe2cf000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1ebe50b000)

[maflcko]

From the closed duplicate issue: A possible workaround is to set -DLIBCXXABI_USE_LLVM_UNWINDER=OFF.

[ldionne]

Is it possible that libunwind is not being built with MemoryWithOrigins even though -DLLVM_USE_SANITIZER=MemoryWithOrigins is passed? Can you check whether libunwind is being built with msan support?

That guess appears to be right on the money!

Nice. So the fix should be pretty clear, if we can fix the CMake to funnel the sanitizer arguments though, it should fix everything.

[ns-osmolsky]

Does the MSan-instrumented build of libunwind work?

[radioflash]

Does the MSan-instrumented build of libunwind work?

No. I think MSAN instrumented libunwind is the actual source of the problem instead. My advice: Dont even build it, because its broken, just use the default, non-MSAN one even for MSAN builds.

From what I see: libunwind DOES get the sanitizer flags when it is rebuilt--but those sanitizer flags actually break libunwind => leading to sanitizer warnings being fired when unwinding (not sure which ones/why exactly). Since libunwind is USED to provide the warnings, this is a recursive problem, and the stack just runs out at some point.

In my view, this is a bug in libunwind that should ideally be fixed there (=> false MSAN positives). Unfortunately, these false positives break MSAN itself, because of recursion.

More Details

I had a similar problem (clang + MSAN + MSAN libc++ just leads to stack overflow). But my case is even simpler:

#include <iostream>

int main()
{
        int x;

        std::cout << "Accessing uninitialized memory..." << std::endl;
        std::cout << x << std::endl;
        std::cout << "Accees to uninitialized memory successful!" << std::endl;

        return x;
}

This crashes with the same output:

Accessing uninitialized memory...
MemorySanitizer:DEADLYSIGNAL
==839299==ERROR: MemorySanitizer: stack-overflow on address 0x7fffcbfd3f00 (pc 0x561f541a7cac bp 0x7fffcbfd4730 sp 0x7fffcbfd3f00 T839299)
MemorySanitizer:DEADLYSIGNAL
MemorySanitizer: nested bug in the same thread, aborting.

I did EVERYTHING completely "by the book": clang-18 from the official package repository (18.1.4) llvm-project at exactly the same version (18.1.4). Then I tried to just use llvm-project to build C++ runtimes with MSAN support, namely "libcxx;libcxxabi;libunwind"

cd llvm-project
mkdir -p build
cmake -GNinja -S runtimes -B build \
      -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_ENABLE_RUNTIMES="libcxx;libcxxabi;libunwind" \
      -DCMAKE_ASM_COMPILER=clang-18 \
      -DCMAKE_C_COMPILER=clang-18 \
      -DCMAKE_CXX_COMPILER=clang++-18 \
      -DLLVM_ENABLE_LLD=YES \
      -DLIBCXX_USE_COMPILER_RT=YES \
      -DLLVM_USE_SANITIZER=MemoryWithOrigins \
      -DCMAKE_INSTALL_PREFIX=/opt/msan
cmake --build build --target install-cxx install-cxxabi install-unwind

These are linked with the super simple test from above:

clang++-18 -O0 -o main.exe main.cpp -fuse-ld=lld -nostdinc++ -nostdlib++ -Wl,-rpath,/opt/msan/lib -L/opt/msan/lib -isystem /opt/msan/include/c++/v1 -lc++ -fPIE -fsanitize=memory -fsanitize-memory-track-origins  && ./main.exe

And that FAILS.

But I can see that the libunwind compilation DID get the msan flags it is supposed to have:

sudo ninja -v install-unwind
[...]
  [2/23] /usr/bin/clang++-18 -D_LIBCPP_HAS_NO_PRAGMA_SYSTEM_HEADER -D_LIBUNWIND_LINK_DL_LIB -D_LIBUNWIND_LINK_PTHREAD_LIB -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/home/w.pupp/llvm-project/libunwind/include -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -Wmisleading-indentation -Wctad-maybe-unsupported -fno-omit-frame-pointer -gline-tables-only -fsanitize=memory -fsanitize-memory-track-origins -fdiagnostics-color -O3 -DNDEBUG -fPIC -Werror=return-type -funwind-tables -nostdinc++ -D_DEBUG -UNDEBUG -D_LIBUNWIND_IS_NATIVE_ONLY -Wall -Wextra -Wnewline-eof -Wshadow -Wwrite-strings -Wno-unused-parameter -Wno-long-long -Werror=return-type -Wextra-semi -Wundef -Wunused-template -Wformat-nonliteral -Wno-user-defined-literals -Wno-covered-switch-default -Wno-suggest-override -Wno-error -pedantic -fno-rtti -std=c++17  -fstrict-aliasing -fno-exceptions -fno-rtti -MD -MT libunwind/src/CMakeFiles/unwind_shared_objects.dir/Unwind-EHABI.cpp.o -MF libunwind/src/CMakeFiles/unwind_shared_objects.dir/Unwind-EHABI.cpp.o.d -o libunwind/src/CMakeFiles/unwind_shared_objects.dir/Unwind-EHABI.cpp.o -c /home/w.pupp/llvm-project/libunwind/src/Unwind-EHABI.cpp
[...]

(Note the -fsanitize=memory -fsanitize-memory-track-origins).

[EricWF]

I just ran into this same issue trying to build a MSAN-enabled version of Clang using libc++.

I can confirm that libunwind was built with MSAN enabled, as it shows MSAN symbols in the library.

Simply deleting the libunwind.so* libraries (which I had with libc++ under /opt/llvm-msan/lib/x86_64-unknown-linux-gnu) resolved the issue, allowing the already built libraries to link a non-msan enabled libunwind.

[SChernykh]

This is still happening in clang 19.1.2