Sanitizers builds are broken

fm4v commented 5 days ago

https://github.com/ClickHouse/ClickHouse/actions/runs/9741485777/job/26881454138?pr=65186

Jul 02 16:51:35 [3090/13667] Linking CXX executable contrib/google-protobuf-cmake/protoc
Jul 02 16:51:35 FAILED: contrib/google-protobuf-cmake/protoc 
Jul 02 16:51:35 : && /usr/bin/clang++-18 --target=x86_64-linux-gnu --sysroot=/build/cmake/linux/../../contrib/sysroot/linux-x86_64/x86_64-linux-gnu/libc --gcc-toolchain=/build/cmake/linux/../../contrib/sysroot/linux-x86_64  -g -fno-omit-frame-pointer -DSANITIZER -fsanitize=address -fsanitize-address-use-after-scope -fdiagnostics-color=always -Xclang -fuse-ctor-homing -Wno-enum-constexpr-conversion -fsized-deallocation  -gdwarf-aranges -pipe -mssse3 -msse4.1 -msse4.2 -mpclmul -mpopcnt -fasynchronous-unwind-tables -ffile-prefix-map=/build=. -ftime-trace -falign-functions=32 -mbranches-within-32B-boundaries -ffp-contract=off  -fdiagnostics-absolute-paths -fstrict-vtable-pointers -w -ffunction-sections -fdata-sections -O2 -g -DNDEBUG -O3 -g --gcc-toolchain=/build/cmake/linux/../../contrib/sysroot/linux-x86_64 --ld-path=/usr/bin/ld.lld-18 -Wl,--no-export-dynamic -Wl,--gc-sections -Wl,--gdb-index -Wl,--build-id=sha1 -Wl,--time-trace contrib/google-protobuf-cmake/CMakeFiles/protoc.dir/__/google-protobuf/src/google/protobuf/compiler/main.cc.o -o contrib/google-protobuf-cmake/protoc  contrib/google-protobuf-cmake/lib_libprotoc.a  contrib/google-protobuf-cmake/lib_libprotobuf.a  -lpthread  contrib/google-protobuf-cmake/libutf8_validity.a  contrib/abseil-cpp-cmake/libabsl_base.a  contrib/abseil-cpp-cmake/libabsl_cord.a  contrib/abseil-cpp-cmake/libabsl_die_if_null.a  contrib/abseil-cpp-cmake/libabsl_hash.a  contrib/abseil-cpp-cmake/libabsl_log_initialize.a  contrib/abseil-cpp-cmake/libabsl_log_severity.a  contrib/abseil-cpp-cmake/libabsl_status.a  contrib/abseil-cpp-cmake/libabsl_statusor.a  contrib/abseil-cpp-cmake/libabsl_strings.a  contrib/abseil-cpp-cmake/libabsl_synchronization.a  contrib/abseil-cpp-cmake/libabsl_time.a  contrib/zlib-ng-cmake/lib_zlib.a  contrib/abseil-cpp-cmake/libabsl_log_internal_check_op.a  contrib/abseil-cpp-cmake/libabsl_leak_check.a  contrib/abseil-cpp-cmake/libabsl_log_internal_conditions.a  contrib/abseil-cpp-cmake/libabsl_log_internal_message.a  contrib/abseil-cpp-cmake/libabsl_log_internal_nullguard.a  contrib/abseil-cpp-cmake/libabsl_examine_stack.a  contrib/abseil-cpp-cmake/libabsl_log_internal_format.a  contrib/abseil-cpp-cmake/libabsl_log_internal_proto.a  contrib/abseil-cpp-cmake/libabsl_log_internal_log_sink_set.a  contrib/abseil-cpp-cmake/libabsl_log_sink.a  contrib/abseil-cpp-cmake/libabsl_log_entry.a  contrib/abseil-cpp-cmake/libabsl_flags_internal.a  contrib/abseil-cpp-cmake/libabsl_flags_marshalling.a  contrib/abseil-cpp-cmake/libabsl_flags_reflection.a  contrib/abseil-cpp-cmake/libabsl_flags_config.a  contrib/abseil-cpp-cmake/libabsl_flags_program_name.a  contrib/abseil-cpp-cmake/libabsl_flags_private_handle_accessor.a  contrib/abseil-cpp-cmake/libabsl_flags_commandlineflag.a  contrib/abseil-cpp-cmake/libabsl_flags_commandlineflag_internal.a  contrib/abseil-cpp-cmake/libabsl_log_globals.a  contrib/abseil-cpp-cmake/libabsl_vlog_config_internal.a  contrib/abseil-cpp-cmake/libabsl_log_internal_fnmatch.a  contrib/abseil-cpp-cmake/libabsl_log_internal_globals.a  contrib/abseil-cpp-cmake/libabsl_raw_hash_set.a  contrib/abseil-cpp-cmake/libabsl_hash.a  contrib/abseil-cpp-cmake/libabsl_city.a  contrib/abseil-cpp-cmake/libabsl_low_level_hash.a  contrib/abseil-cpp-cmake/libabsl_hashtablez_sampler.a  contrib/abseil-cpp-cmake/libabsl_status.a  contrib/abseil-cpp-cmake/libabsl_cord.a  contrib/abseil-cpp-cmake/libabsl_cordz_info.a  contrib/abseil-cpp-cmake/libabsl_cord_internal.a  contrib/abseil-cpp-cmake/libabsl_cordz_functions.a  contrib/abseil-cpp-cmake/libabsl_exponential_biased.a  contrib/abseil-cpp-cmake/libabsl_cordz_handle.a  contrib/abseil-cpp-cmake/libabsl_crc_cord_state.a  contrib/abseil-cpp-cmake/libabsl_crc32c.a  contrib/abseil-cpp-cmake/libabsl_crc_internal.a  contrib/abseil-cpp-cmake/libabsl_crc_cpu_detect.a  contrib/abseil-cpp-cmake/libabsl_bad_optional_access.a  contrib/abseil-cpp-cmake/libabsl_strerror.a  contrib/abseil-cpp-cmake/libabsl_str_format_internal.a  contrib/abseil-cpp-cmake/libabsl_synchronization.a  contrib/abseil-cpp-cmake/libabsl_stacktrace.a  contrib/abseil-cpp-cmake/libabsl_symbolize.a  contrib/abseil-cpp-cmake/libabsl_debugging_internal.a  contrib/abseil-cpp-cmake/libabsl_demangle_internal.a  contrib/abseil-cpp-cmake/libabsl_graphcycles_internal.a  contrib/abseil-cpp-cmake/libabsl_kernel_timeout_internal.a  contrib/abseil-cpp-cmake/libabsl_malloc_internal.a  contrib/abseil-cpp-cmake/libabsl_time.a  contrib/abseil-cpp-cmake/libabsl_strings.a  contrib/abseil-cpp-cmake/libabsl_strings_internal.a  contrib/abseil-cpp-cmake/libabsl_string_view.a  contrib/abseil-cpp-cmake/libabsl_base.a  contrib/abseil-cpp-cmake/libabsl_spinlock_wait.a  contrib/abseil-cpp-cmake/libabsl_throw_delegate.a  contrib/abseil-cpp-cmake/libabsl_int128.a  contrib/abseil-cpp-cmake/libabsl_civil_time.a  contrib/abseil-cpp-cmake/libabsl_time_zone.a  contrib/abseil-cpp-cmake/libabsl_bad_variant_access.a  contrib/abseil-cpp-cmake/libabsl_raw_logging_internal.a  contrib/abseil-cpp-cmake/libabsl_log_severity.a  -Wl,--start-group  contrib/libcxx-cmake/libcxx.a  contrib/libcxxabi-cmake/libcxxabi.a  contrib/libunwind-cmake/libunwind.a  base/glibc-compatibility/libglibc-compatibility.a  base/glibc-compatibility/memcpy/libmemcpy.a  -Wl,--end-group  -nodefaultlibs -lgcc   -lc -lm -lrt -lpthread -ldl && :
Jul 02 16:51:35 ld.lld-18: error: cannot open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan_static-x86_64.a: No such file or directory
Jul 02 16:51:35 ld.lld-18: error: cannot open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a: No such file or directory
Jul 02 16:51:35 ld.lld-18: error: cannot open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan_cxx-x86_64.a: No such file or directory
Jul 02 16:51:35 clang++-18: error: linker command failed with exit code 1 (use -v to see invocation)

In docker container:

find /usr/lib -name "libclang_rt.asan*.a"

/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan-preinit.a
/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a
/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan_cxx.a
/usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan_static.a
/usr/lib/llvm-18/lib/clang/18/lib/i386-pc-linux-gnu/libclang_rt.asan-preinit.a
/usr/lib/llvm-18/lib/clang/18/lib/i386-pc-linux-gnu/libclang_rt.asan.a
/usr/lib/llvm-18/lib/clang/18/lib/i386-pc-linux-gnu/libclang_rt.asan_cxx.a
/usr/lib/llvm-18/lib/clang/18/lib/i386-pc-linux-gnu/libclang_rt.asan_static.a

azat commented 5 days ago

I guess this is what happened - you PR (https://github.com/ClickHouse/ClickHouse/pull/65186) updated the fasttest/run.sh, which should lead to updating the docker image, which updates the clang compiler, which by some reason, moved rt libraries under linux:

$ docker run --rm -it clickhouse/binary-builder clang-18 --version
Ubuntu clang version 18.1.6 (++20240517093811+3d0752b9492e-1~exp1~20240517213934.128)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

# your pr!
$ docker run --rm -it clickhouse/binary-builder:9e5aa3a43749 clang-18 --version
Ubuntu clang version 18.1.8 (++20240615103753+3b5b5c1ec4a3-1~exp1~20240615223858.136)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

I've seen this quite a few times, that clang/llvm changes their paths in the packages, and usually I'm just adding some quirks into the Dockerfile:

# This symlink is required by gcc to find the lld linker
RUN ln -s /usr/bin/lld-${LLVM_VERSION} /usr/bin/ld.lld
# FIXME: workaround for "The imported target "merge-fdata" references the file" error
# https://salsa.debian.org/pkg-llvm-team/llvm-toolchain/-/commit/992e52c0b156a5ba9c6a8a54f8c4857ddd3d371d
RUN sed -i '/_IMPORT_CHECK_FILES_FOR_\(mlir-\|llvm-bolt\|merge-fdata\|MLIR\)/ {s|^|#|}' /usr/lib/llvm-${LLVM_VERSION}/lib/cmake/llvm/LLVMExports-*.cmake

But actually this will not help, the problem is ccache I guess

So I would rather recommend you to pin the compiler version for now (and actually I want to do this long time ago, since this implicit updates is odd)

azat commented 5 days ago

FWIW clang-18.1.8 changes the location of runtime libraries:

# 18.1.8
root@a6a45ab18ca8:/workdir# clang-18 -Wl,--verbose -fsanitize=address |& grep libclang_rt |& grep succ
attempt to open /usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan_static.a succeeded
attempt to open /usr/lib/llvm-18/lib/clang/18/lib/x86_64-pc-linux-gnu/libclang_rt.asan.a succeeded

# 18.1.6
root@71d08be89248:/workdir# clang-18 -Wl,--verbose -fsanitize=address |& grep libclang_rt |& grep succ
attempt to open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan_static-x86_64.a succeeded
attempt to open /usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.a succeeded

And sanitizers does works in 18.1.8

root@f8b02eee88c1:/workdir# clang-18 --version
Ubuntu clang version 18.1.8 (++20240615103753+3b5b5c1ec4a3-1~exp1~20240615223858.136)
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

root@a6a45ab18ca8:/workdir# echo 'int main() { return 0; }' | clang-18 -xc - -fsanitize=address -o /tmp/main && /tmp/main && echo OK && ldd /tmp/main | grep san
OK

root@f8b02eee88c1:/workdir# ulimit -n 100
root@f8b02eee88c1:/workdir# echo -e '#include <stdlib.h>\nint main() { malloc(10); }' | clang-18 -xc - -fsanitize=address -o /tmp/main && /tmp/main && echo OK

=================================================================
==13==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 10 byte(s) in 1 object(s) allocated from:
    #0 0x55555561957f in malloc (/tmp/main+0xc557f) (BuildId: 422714ff3930e4a95e0ef582fa5ea71eb602f7f6)
    #1 0x55555565686d in main (/tmp/main+0x10286d) (BuildId: 422714ff3930e4a95e0ef582fa5ea71eb602f7f6)
    #2 0x7ffff7c9bd8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)

SUMMARY: AddressSanitizer: 10 byte(s) leaked in 1 allocation(s).

Also note, that rt libraries are static there as well:

root@f8b02eee88c1:/workdir# ldd /tmp/main
        linux-vdso.so.1 (0x00007ffff7fc1000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffff743d000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007ffff7429000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007ffff7409000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffff71e0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffff7fc3000)

azat commented 5 days ago

P.S. I can't find updating of fasttest (and neither binary-builder) image here - https://play.clickhouse.com/play?user=play#U0VMRUNUIGNoZWNrX3N0YXJ0X3RpbWUsIGNoZWNrX25hbWUsIHRlc3RfbmFtZSwgcmVwb3J0X3VybApGUk9NIGNoZWNrcwpXSEVSRSByZXBvcnRfdXJsIExJS0UgJyVkb2NrZXIlJwogICAgQU5EIGNoZWNrX3N0YXJ0X3RpbWUgPj0gbm93KCkgLSBJTlRFUlZBTCAyNCBkYXkKICAgIEFORCBwdWxsX3JlcXVlc3RfbnVtYmVyID0gNjUxODYKT1JERVIgQlkgY2hlY2tfbmFtZSwgdGVzdF9uYW1lLCBjaGVja19zdGFydF90aW1l

But maybe there is some issues with this database or/and descriptions/urls