search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
8.91k stars 1.79k forks source link

heap-use-after-free when killing BE with ASAN mode #6941

Closed ZiheLiu closed 1 year ago

ZiheLiu commented 2 years ago

Steps to reproduce the behavior (Required)

  1. Start BE.
  2. kill -15 <be-pid>. Send SIGTERM signal to BE.

Expected behavior (Required)

BE shutdowns normally.

Real behavior (Required)

ExternalScanContextMgr

  1. The thread ExternalScanContextMgr::_keep_alive_reaper started in ExecEnv::init().
  2. ExternalScanContextMgr is desctucted by ExecEnv::destroy() when BE shutdowning.
  3. ExternalScanContextMgr::_keep_alive_reaper is a detach thread and still running, and using the member variables from ExternalScanContextMgr.
=================================================================
==189249==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b0000bf1b0 at pc 0x00000632bb83 bp 0x7f9c2a9becd0 sp 0x7f9c2a9becc8
READ of size 8 at 0x60b0000bf1b0 thread T136
    #22 0x7f9a97e9911f  (<unknown module>)
    #0 0x632bb82 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> > > >::begin() /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/stl_tree.h:1003
    #23 0x7f9a97e9911f  (<unknown module>)
    #1 0x632a589 in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<starrocks::ScanContext>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> > > >::begin() /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/stl_map.h:357
    #24 0x7f9a97e98b6f  (<unknown module>)
    #2 0x6327470 in starrocks::ExternalScanContextMgr::gc_expired_context() /home/disk2/liuzihe/starrocks/be/src/runtime/external_scan_context_mgr.cpp:110
    #25 0x7f9a97e98b6f  (<unknown module>)
    #3 0x632f544 in void std::__invoke_impl<void, void (starrocks::ExternalScanContextMgr::* const&)(), starrocks::ExternalScanContextMgr*&>(std::__invoke_memfun_deref, void (starrocks::ExternalScanContextMgr::* const&)(), starrocks::ExternalScanContextMgr*&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:73
    #26 0x7f9a97e98bb4  (<unknown module>)
    #4 0x632f3d9 in std::__invoke_result<void (starrocks::ExternalScanContextMgr::* const&)(), starrocks::ExternalScanContextMgr*&>::type std::__invoke<void (starrocks::ExternalScanContextMgr::* const&)(), starrocks::ExternalScanContextMgr*&>(void (starrocks::ExternalScanContextMgr::* const&)(), starrocks::ExternalScanContextMgr*&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:95
    #27 0x7f9a97e917ca  (<unknown module>)
    #5 0x632f39f in decltype (__invoke((*this)._M_pmf, (forward<starrocks::ExternalScanContextMgr*&>)({parm#1}))) std::_Mem_fn_base<void (starrocks::ExternalScanContextMgr::*)(), true>::operator()<starrocks::ExternalScanContextMgr*&>(starrocks::ExternalScanContextMgr*&) const /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/functional:122
    #28 0x7f9c5b05d39a in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x68839a)
    #6 0x632f36a in void std::__invoke_impl<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()>&, starrocks::ExternalScanContextMgr*&>(std::__invoke_other, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()>&, starrocks::ExternalScanContextMgr*&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
    #29 0x7f9c5b0a8350 in jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) [clone .isra.96] [clone .constprop.117] (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x6d3350)

Direct leak of 120 byte(s) in 1 object(s) allocated from:
    #7 0x632f2f6 in std::enable_if<is_invocable_r_v<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()>&, starrocks::ExternalScanContextMgr*&>, void>::type std::__invoke_r<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()>&, starrocks::ExternalScanContextMgr*&>(std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()>&, starrocks::ExternalScanContextMgr*&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:110
    #8 0x632f233 in void std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/functional:566
    #0 0x4aed8ff in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:145
    #9 0x632f129 in void std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>::operator()<>() /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/functional:625
    #1 0x7f9c5b2da454 in os::malloc(unsigned long, MemoryType, NativeCallStack const&) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x905454)
    #10 0x632f081 in void std::__invoke_impl<void, std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>>(std::__invoke_other, std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>&&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
    #2 0x7f9c5ac9bf48 in CHeapObj<(MemoryType)7>::operator new(unsigned long, NativeCallStack const&) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x2c6f48)
    #3 0x7f9c5ac9bfe3 in CHeapObj<(MemoryType)7>::operator new(unsigned long) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x2c6fe3)
    #4 0x7f9c5aeb12f3 in ConstantPool::allocate(ClassLoaderData*, int, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x4dc2f3)
    #11 0x632f036 in std::__invoke_result<std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>>::type std::__invoke<std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>>(std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>&&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:95
    #5 0x7f9c5ae0b368 in ClassFileParser::parse_constant_pool(Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x436368)
    #6 0x7f9c5ae0c6f6 in ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>*, TempNewSymbol&, bool, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x4376f6)
    #12 0x632efe3 in void std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/thread:264
    #7 0x7f9c5ae13e12 in ClassLoader::load_classfile(Symbol*, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x43ee12)
    #13 0x632efb7 in std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)> > >::operator()() /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/thread:271
    #8 0x7f9c5b419cae in SystemDictionary::load_instance_class(Symbol*, Handle, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0xa44cae)
    #14 0x632ef9b in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)> > > >::_M_run() /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/thread:215
    #9 0x7f9c5b41871a in SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0xa4371a)
    #15 0xce2780f in execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:80
    #10 0x7f9c5b41a5f8 in SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0xa455f8)
    #16 0x7f9c5a7c0ea4 in start_thread (/lib64/libpthread.so.0+0x7ea4)
    #11 0x7f9c5aea4b96 in ConstantPool::klass_at_impl(constantPoolHandle, int, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x4cfb96)
    #17 0x7f9c59ddbb0c in clone (/lib64/libc.so.6+0xfeb0c)

    #12 0x7f9c5b04b13c in InterpreterRuntime::anewarray(JavaThread*, ConstantPool*, int, int) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x67613c)
0x60b0000bf1b0 is located 32 bytes inside of 104-byte region [0x60b0000bf190,0x60b0000bf1f8)
freed by thread T0 here:
    #13 0x7f9a97eb8f55  (<unknown module>)
    #0 0x4aefed7 in operator delete(void*, unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:172
    #14 0x7f9a97e993d5  (<unknown module>)
    #1 0x61cc4ba in starrocks::ExecEnv::_destroy() /home/disk2/liuzihe/starrocks/be/src/runtime/exec_env.cpp:516
    #15 0x7f9a97e993d5  (<unknown module>)
    #2 0x61cc53b in starrocks::ExecEnv::destroy(starrocks::ExecEnv*) /home/disk2/liuzihe/starrocks/be/src/runtime/exec_env.cpp:523
    #16 0x7f9a97e9911f  (<unknown module>)
    #3 0x4b2fe5a in main /home/disk2/liuzihe/starrocks/be/src/service/starrocks_main.cpp:297
    #17 0x7f9a97e9911f  (<unknown module>)
    #4 0x7f9c59cff554 in __libc_start_main (/lib64/libc.so.6+0x22554)

previously allocated by thread T0 here:
    #18 0x7f9a97e9911f  (<unknown module>)
    #0 0x4aef0b7 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #19 0x7f9a97e9911f  (<unknown module>)
    #1 0x61c3dc5 in starrocks::ExecEnv::_init(std::vector<starrocks::StorePath, std::allocator<starrocks::StorePath> > const&) /home/disk2/liuzihe/starrocks/be/src/runtime/exec_env.cpp:124
    #20 0x7f9a97e9911f  (<unknown module>)
    #2 0x61c388d in starrocks::ExecEnv::init(starrocks::ExecEnv*, std::vector<starrocks::StorePath, std::allocator<starrocks::StorePath> > const&) /home/disk2/liuzihe/starrocks/be/src/runtime/exec_env.cpp:119
    #21 0x7f9a97e9911f  (<unknown module>)
    #3 0x4b2f46b in main /home/disk2/liuzihe/starrocks/be/src/service/starrocks_main.cpp:248
    #22 0x7f9a97e9911f  (<unknown module>)
    #4 0x7f9c59cff554 in __libc_start_main (/lib64/libc.so.6+0x22554)

Thread T136 created by T0 here:
    #23 0x7f9a97e9911f  (<unknown module>)
    #0 0x4a98db2 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:214
    #24 0x7f9a97e9911f  (<unknown module>)
    #1 0xce278d4 in __gthread_create /home/disk2/zc/tools/toolchain/build/gcc-10.3.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663
    #2 0xce278d4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:135
    #25 0x7f9a97e993d5  (<unknown module>)
    #3 0x6329adf in std::_MakeUniq<std::thread>::__single_object std::make_unique<std::thread, std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)> >(std::_Bind_result<void, std::_Mem_fn<void (starrocks::ExternalScanContextMgr::*)()> (starrocks::ExternalScanContextMgr*)>&&) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/unique_ptr.h:962
    #26 0x7f9a97e917ca  (<unknown module>)
    #4 0x6325975 in starrocks::ExternalScanContextMgr::ExternalScanContextMgr(starrocks::ExecEnv*) /home/disk2/liuzihe/starrocks/be/src/runtime/external_scan_context_mgr.cpp:39
    #27 0x7f9c5b05d39a in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x68839a)
    #5 0x61c3dda in starrocks::ExecEnv::_init(std::vector<starrocks::StorePath, std::allocator<starrocks::StorePath> > const&) /home/disk2/liuzihe/starrocks/be/src/runtime/exec_env.cpp:124
    #28 0x7f9c5b019083 in InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x644083)
    #6 0x61c388d in starrocks::ExecEnv::init(starrocks::ExecEnv*, std::vector<starrocks::StorePath, std::allocator<starrocks::StorePath> > const&) /home/disk2/liuzihe/starrocks/be/src/runtime/exec_env.cpp:119
    #29 0x7f9c5b01946d in InstanceKlass::initialize_impl(instanceKlassHandle, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x64446d)

    #7 0x4b2f46b in main /home/disk2/liuzihe/starrocks/be/src/service/starrocks_main.cpp:248
    #8 0x7f9c59cff554 in __libc_start_main (/lib64/libc.so.6+0x22554)

Direct leak of 120 byte(s) in 1 object(s) allocated from:
SUMMARY: AddressSanitizer: heap-use-after-free /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/stl_tree.h:1003 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<starrocks::ScanContext> > > >::begin()
    #0 0x4aed8ff in __interceptor_malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:145
    #1 0x7f9c5b2da454 in os::malloc(unsigned long, MemoryType, NativeCallStack const&) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x905454)
    #2 0x7f9c5ac9bf48 in CHeapObj<(MemoryType)7>::operator new(unsigned long, NativeCallStack const&) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x2c6f48)
    #3 0x7f9c5ac9bfe3 in CHeapObj<(MemoryType)7>::operator new(unsigned long) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x2c6fe3)
    #4 0x7f9c5aeb12f3 in ConstantPool::allocate(ClassLoaderData*, int, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x4dc2f3)
    #5 0x7f9c5ae0b368 in ClassFileParser::parse_constant_pool(Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x436368)
    #6 0x7f9c5ae0c6f6 in ClassFileParser::parseClassFile(Symbol*, ClassLoaderData*, Handle, KlassHandle, GrowableArray<Handle>*, TempNewSymbol&, bool, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x4376f6)
    #7 0x7f9c5ae13e12 in ClassLoader::load_classfile(Symbol*, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x43ee12)
    #8 0x7f9c5b419cae in SystemDictionary::load_instance_class(Symbol*, Handle, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0xa44cae)
    #9 0x7f9c5b41871a in SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0xa4371a)
    #10 0x7f9c5b41a5f8 in SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0xa455f8)
    #11 0x7f9c5aea4b96 in ConstantPool::klass_at_impl(constantPoolHandle, int, Thread*) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x4cfb96)
    #12 0x7f9c5b04a9b2 in InterpreterRuntime::_new(JavaThread*, ConstantPool*, int) (/home/disk1/doris-deps/toolchain/installed/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so+0x6759b2)
Shadow bytes around the buggy address:
  0x0c168000fde0: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
  0x0c168000fdf0: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c168000fe00: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x0c168000fe10: fd fd fd fa fa fa fa fa fa fa fa fa fd fd fd fd
  0x0c168000fe20: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa
=>0x0c168000fe30: fa fa fd fd fd fd[fd]fd fd fd fd fd fd fd fd fa
  0x0c168000fe40: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c168000fe50: fd fd fd fd fd fa fa fa fa fa fa fa fa fa fd fd
  0x0c168000fe60: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
  0x0c168000fe70: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c168000fe80: fd fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==189249==ABORTING

2. EvHttpServer

  1. The thread EvHttpServer::worker is created in start_be().
  2. EvHttpServer will be desctructed by http_service.reset() in start_be() after receiving SIGTERM.
  3. The thread EvHttpServer::worker is detach and and still running, and using the member variables from EvHttpServer.
=================================================================
==21088==ERROR: AddressSanitizer: heap-use-after-free on address 0x6190004aead0 at pc 0x000006f7c94c bp 0x7fd2a9c8da90 sp 0x7fd2a9c8da88
READ of size 1 at 0x6190004aead0 thread T550
    #0 0x6f7c94b in starrocks::PathTrie<starrocks::HttpHandler*>::split(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*) /home/disk2/liuzihe/starrocks/be/src/util/path_trie.hpp:270
    #1 0x6f7bee8 in starrocks::PathTrie<starrocks::HttpHandler*>::retrieve(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, starrocks::HttpHandler**, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >*) /home/disk2/liuzihe/starrocks/be/src/util/path_trie.hpp:248
    #2 0x6f764be in starrocks::EvHttpServer::_find_handler(starrocks::HttpRequest*) /home/disk2/liuzihe/starrocks/be/src/http/ev_http_server.cpp:278
    #3 0x6f75e8b in starrocks::EvHttpServer::on_header(evhttp_request*) /home/disk2/liuzihe/starrocks/be/src/http/ev_http_server.cpp:228
    #4 0x6f72d49 in on_header /home/disk2/liuzihe/starrocks/be/src/http/ev_http_server.cpp:67
    #5 0xb3274d9 in evhttp_read_header /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/http.c:2279
    #6 0xb329992 in bufferevent_trigger_nolock_ /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/bufferevent-internal.h:411
    #7 0xb329992 in bufferevent_readcb /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/bufferevent_sock.c:219
    #8 0xb317ce2 in event_persist_closure /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/event.c:1608
    #9 0xb317ce2 in event_process_active_single_queue /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/event.c:1667
    #10 0xb3183c6 in event_process_active /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/event.c:1768
    #11 0xb3183c6 in event_base_loop /home/disk2/zc/incubator-doris/thirdparty/src/libevent-master/event.c:1991
    #12 0x6f73f51 in operator() /home/disk2/liuzihe/starrocks/be/src/http/ev_http_server.cpp:122
    #13 0x6f7b061 in __invoke_impl<void, starrocks::EvHttpServer::start()::<lambda()> > /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:60
    #14 0x6f7b008 in __invoke<starrocks::EvHttpServer::start()::<lambda()> > /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/invoke.h:95
    #15 0x6f7af99 in _M_invoke<0> /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/thread:264
    #16 0x6f7aee9 in operator() /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/thread:271
    #17 0x6f7aa25 in _M_run /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/thread:215
    #18 0xce2780f in execute_native_thread_routine ../../../../../libstdc++-v3/src/c++11/thread.cc:80
    #19 0x7fd807b22e64 in start_thread (/lib64/libpthread.so.0+0x7e64)
    #20 0x7fd80713d88c in clone (/lib64/libc.so.6+0xfe88c)

0x6190004aead0 is located 848 bytes inside of 1000-byte region [0x6190004ae780,0x6190004aeb68)
freed by thread T0 here:
    #0 0x4aefed7 in operator delete(void*, unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:172
    #1 0x64f3ffe in std::default_delete<starrocks::EvHttpServer>::operator()(starrocks::EvHttpServer*) const /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/unique_ptr.h:85
    #2 0x64f42db in std::__uniq_ptr_impl<starrocks::EvHttpServer, std::default_delete<starrocks::EvHttpServer> >::reset(starrocks::EvHttpServer*) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/unique_ptr.h:182
    #3 0x64f22ec in std::unique_ptr<starrocks::EvHttpServer, std::default_delete<starrocks::EvHttpServer> >::reset(starrocks::EvHttpServer*) /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/unique_ptr.h:456
    #4 0x64eb120 in starrocks::HttpServiceBE::~HttpServiceBE() /home/disk2/liuzihe/starrocks/be/src/service/service_be/http_service.cpp:57
    #5 0x64e1555 in std::default_delete<starrocks::HttpServiceBE>::operator()(starrocks::HttpServiceBE*) const (/home/disk1/sr/pipeline_benchmark/be/lib/starrocks_be+0x64e1555)
    #6 0x64e161f in std::__uniq_ptr_impl<starrocks::HttpServiceBE, std::default_delete<starrocks::HttpServiceBE> >::reset(starrocks::HttpServiceBE*) (/home/disk1/sr/pipeline_benchmark/be/lib/starrocks_be+0x64e161f)
    #7 0x64e113c in std::unique_ptr<starrocks::HttpServiceBE, std::default_delete<starrocks::HttpServiceBE> >::reset(starrocks::HttpServiceBE*) (/home/disk1/sr/pipeline_benchmark/be/lib/starrocks_be+0x64e113c)
    #8 0x64dee0d in start_be() /home/disk2/liuzihe/starrocks/be/src/service/service_be/starrocks_be.cpp:74
    #9 0x4b2fc9f in main /home/disk2/liuzihe/starrocks/be/src/service/starrocks_main.cpp:282
    #10 0x7fd807061504 in __libc_start_main (/lib64/libc.so.6+0x22504)

previously allocated by thread T0 here:
    #0 0x4aef0b7 in operator new(unsigned long) ../../../../libsanitizer/asan/asan_new_delete.cpp:99
    #1 0x64eb032 in starrocks::HttpServiceBE::HttpServiceBE(starrocks::ExecEnv*, int, int) /home/disk2/liuzihe/starrocks/be/src/service/service_be/http_service.cpp:53
    #2 0x64e0f63 in std::_MakeUniq<starrocks::HttpServiceBE>::__single_object std::make_unique<starrocks::HttpServiceBE, starrocks::ExecEnv*&, int&, int&>(starrocks::ExecEnv*&, int&, int&) (/home/disk1/sr/pipeline_benchmark/be/lib/starrocks_be+0x64e0f63)
    #3 0x64deb05 in start_be() /home/disk2/liuzihe/starrocks/be/src/service/service_be/starrocks_be.cpp:59
    #4 0x4b2fc9f in main /home/disk2/liuzihe/starrocks/be/src/service/starrocks_main.cpp:282
    #5 0x7fd807061504 in __libc_start_main (/lib64/libc.so.6+0x22504)

Thread T550 created by T0 here:
    #0 0x4a98db2 in __interceptor_pthread_create ../../../../libsanitizer/asan/asan_interceptors.cpp:214
    #1 0xce278d4 in __gthread_create /home/disk2/zc/tools/toolchain/build/gcc-10.3.0/build/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:663
    #2 0xce278d4 in std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State> >, void (*)()) ../../../../../libstdc++-v3/src/c++11/thread.cc:135
    #3 0x6f772a6 in construct<std::thread, starrocks::EvHttpServer::start()::<lambda()>&> /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/ext/new_allocator.h:150
    #4 0x6f76bbb in construct<std::thread, starrocks::EvHttpServer::start()::<lambda()>&> /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/alloc_traits.h:512
    #5 0x6f7689d in emplace_back<starrocks::EvHttpServer::start()::<lambda()>&> /home/disk1/doris-deps/toolchain/installed/gcc-10.3.0/include/c++/10.3.0/bits/vector.tcc:115
    #6 0x6f7439e in starrocks::EvHttpServer::start() /home/disk2/liuzihe/starrocks/be/src/http/ev_http_server.cpp:124
    #7 0x64ef167 in starrocks::HttpServiceBE::start() /home/disk2/liuzihe/starrocks/be/src/service/service_be/http_service.cpp:193
    #8 0x64deb61 in start_be() /home/disk2/liuzihe/starrocks/be/src/service/service_be/starrocks_be.cpp:60
    #9 0x4b2fc9f in main /home/disk2/liuzihe/starrocks/be/src/service/starrocks_main.cpp:282
    #10 0x7fd807061504 in __libc_start_main (/lib64/libc.so.6+0x22504)

SUMMARY: AddressSanitizer: heap-use-after-free /home/disk2/liuzihe/starrocks/be/src/util/path_trie.hpp:270 in starrocks::PathTrie<starrocks::HttpHandler*>::split(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)
Shadow bytes around the buggy address:
  0x0c328008dd00: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c328008dd10: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c328008dd20: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c328008dd30: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c328008dd40: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c328008dd50: fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd fd
  0x0c328008dd60: fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa
  0x0c328008dd70: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c328008dd80: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c328008dd90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c328008dda0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==21088==ABORTING

StarRocks version (Required)

github-actions[bot] commented 1 year ago

We have marked this issue as stale because it has been inactive for 6 months. If this issue is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 10 days to keep the issue queue tidy. Thank you for your contribution to StarRocks!