search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
9.02k stars 1.08k forks source link

[DocDB] ANNMethodKind/VectorLSMTest.Simple/kHnswlib fails in TSAN #25017

Open spolitov opened 1 day ago

spolitov commented 1 day ago

Jira Link: DB-14158

Description

WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=196916)
  Cycle in lock order graph: M0 (0x7bb400000000) => M1 (0x7b5000000690) => M0

  Mutex M1 acquired here while holding mutex M0 in thread T1:
    #0 pthread_mutex_lock ${YB_LLVM_TOOLCHAIN_DIR}-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1339:3 (vector_lsm-test+0xacc6b)
I1120 18:50:40.828621 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #1 std::mutex::lock() <null> (libc++.so.1+0xa7749)
    #2 std::unique_lock<std::mutex>::unique_lock[abi:ue170006](std::mutex&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__mutex/unique_lock.h:41:11 (libvector_index.so+0x6718f)
    #3 hnswlib::HierarchicalNSW<float>::addPoint(void const*, unsigned long, int) ${YB_SRC_ROOT}/src/inline-thirdparty/hnswlib/hnswlib/hnswalg.h:1234:39 (libvector_index.so+0x6718f)
    #4 hnswlib::HierarchicalNSW<float>::addPoint(void const*, unsigned long, bool) ${YB_SRC_ROOT}/src/inline-thirdparty/hnswlib/hnswlib/hnswalg.h:1003:13 (libvector_index.so+0x64739)
I1120 18:50:40.828712 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #5 yb::vector_index::(anonymous namespace)::HnswlibIndex<std::vector<float, std::allocator<float>>, float>::DoInsert(unsigned long, std::vector<float, std::allocator<float>> const&) ${YB_SRC_ROOT}/src/yb/vector_index/hnswlib_wrapper.cc:94:12 (libvector_index.so+0x58e6e)
    #6 yb::vector_index::IndexWrapperBase<yb::vector_index::(anonymous namespace)::HnswlibIndex<std::vector<float, std::allocator<float>>, float>, std::vector<float, std::allocator<float>>, float>::Insert(unsigned long, std::vector<float, std::allocator<float>> const&) ${YB_SRC_ROOT}/src/yb/vector_index/index_wrapper_base.h:34:5 (libvector_index.so+0x58e6e)
    #7 non-virtual thunk to yb::vector_index::IndexWrapperBase<yb::vector_index::(anonymous namespace)::HnswlibIndex<std::vector<float, std::allocator<float>>, float>, std::vector<float, std::allocator<float>>, float>::Insert(unsigned long, std::vector<float, std::allocator<float>> const&) ${YB_SRC_ROOT}/src/yb/vector_index/index_wrapper_base.h (libvector_index.so+0x596f4)
I1120 18:50:40.828804 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #8 yb::vector_index::VectorLSMInsertTask<std::vector<float, std::allocator<float>>, float>::Run() ${YB_SRC_ROOT}/src/yb/vector_index/vector_lsm.cc:102:40 (libvector_index.so+0xbb12b)
    #9 yb::rpc::(anonymous namespace)::Worker::Execute() ${YB_SRC_ROOT}/src/yb/rpc/thread_pool.cc:115:15 (libyrpc.so+0x1beb44)
    #10 decltype(*std::declval<yb::rpc::(anonymous namespace)::Worker*&>().*std::declval<void (yb::rpc::(anonymous namespace)::Worker::*&)()>()()) std::__invoke[abi:ue170006]<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:308:25 (libyrpc.so+0x1befcb)
    #11 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__apply_functor[abi:ue170006]<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, std::tuple<>>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__tuple_indices<0ul>, std::tuple<>&&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:260:12 (libyrpc.so+0x1befcb)
I1120 18:50:40.828902 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #12 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()[abi:ue170006]<>() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:292:20 (libyrpc.so+0x1befcb)
    #13 decltype(std::declval<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>()()) std::__invoke[abi:ue170006]<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:340:25 (libyrpc.so+0x1befcb)
I1120 18:50:40.829001 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #14 void std::__invoke_void_return_wrapper<void, true>::__call[abi:ue170006]<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:415:5 (libyrpc.so+0x1befcb)
    #15 std::__function::__alloc_func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()[abi:ue170006]() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:192:16 (libyrpc.so+0x1befcb)
    #16 std::__function::__func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:363:12 (libyrpc.so+0x1befcb)
I1120 18:50:40.829090 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #17 std::__function::__value_func<void ()>::operator()[abi:ue170006]() const ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:517:16 (libyb_util.so+0x43615f)
    #18 std::function<void ()>::operator()() const ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1168:12 (libyb_util.so+0x43615f)
    #19 yb::Thread::SuperviseThread(void*) ${YB_SRC_ROOT}/src/yb/util/thread.cc:888:3 (libyb_util.so+0x43615f)

  Mutex M0 previously acquired by the same thread here:
    #0 pthread_mutex_lock ${YB_LLVM_TOOLCHAIN_DIR}-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1339:3 (vector_lsm-test+0xacc6b)
I1120 18:50:40.829195 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #1 std::mutex::lock() <null> (libc++.so.1+0xa7749)
    #2 std::unique_lock<std::mutex>::unique_lock[abi:ue170006](std::mutex&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__mutex/unique_lock.h:41:11 (libvector_index.so+0x6702e)
    #3 hnswlib::HierarchicalNSW<float>::addPoint(void const*, unsigned long, int) ${YB_SRC_ROOT}/src/inline-thirdparty/hnswlib/hnswlib/hnswalg.h:1227:39 (libvector_index.so+0x6702e)
    #4 hnswlib::HierarchicalNSW<float>::addPoint(void const*, unsigned long, bool) ${YB_SRC_ROOT}/src/inline-thirdparty/hnswlib/hnswlib/hnswalg.h:1003:13 (libvector_index.so+0x64739)
I1120 18:50:40.829285 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #5 yb::vector_index::(anonymous namespace)::HnswlibIndex<std::vector<float, std::allocator<float>>, float>::DoInsert(unsigned long, std::vector<float, std::allocator<float>> const&) ${YB_SRC_ROOT}/src/yb/vector_index/hnswlib_wrapper.cc:94:12 (libvector_index.so+0x58e6e)
    #6 yb::vector_index::IndexWrapperBase<yb::vector_index::(anonymous namespace)::HnswlibIndex<std::vector<float, std::allocator<float>>, float>, std::vector<float, std::allocator<float>>, float>::Insert(unsigned long, std::vector<float, std::allocator<float>> const&) ${YB_SRC_ROOT}/src/yb/vector_index/index_wrapper_base.h:34:5 (libvector_index.so+0x58e6e)
    #7 non-virtual thunk to yb::vector_index::IndexWrapperBase<yb::vector_index::(anonymous namespace)::HnswlibIndex<std::vector<float, std::allocator<float>>, float>, std::vector<float, std::allocator<float>>, float>::Insert(unsigned long, std::vector<float, std::allocator<float>> const&) ${YB_SRC_ROOT}/src/yb/vector_index/index_wrapper_base.h (libvector_index.so+0x596f4)
I1120 18:50:40.829375 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #8 yb::vector_index::VectorLSMInsertTask<std::vector<float, std::allocator<float>>, float>::Run() ${YB_SRC_ROOT}/src/yb/vector_index/vector_lsm.cc:102:40 (libvector_index.so+0xbb12b)
    #9 yb::rpc::(anonymous namespace)::Worker::Execute() ${YB_SRC_ROOT}/src/yb/rpc/thread_pool.cc:115:15 (libyrpc.so+0x1beb44)
    #10 decltype(*std::declval<yb::rpc::(anonymous namespace)::Worker*&>().*std::declval<void (yb::rpc::(anonymous namespace)::Worker::*&)()>()()) std::__invoke[abi:ue170006]<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:308:25 (libyrpc.so+0x1befcb)
    #11 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__apply_functor[abi:ue170006]<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, std::tuple<>>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__tuple_indices<0ul>, std::tuple<>&&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:260:12 (libyrpc.so+0x1befcb)
I1120 18:50:40.829483 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #12 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()[abi:ue170006]<>() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:292:20 (libyrpc.so+0x1befcb)
    #13 decltype(std::declval<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>()()) std::__invoke[abi:ue170006]<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:340:25 (libyrpc.so+0x1befcb)
I1120 18:50:40.829618 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #14 void std::__invoke_void_return_wrapper<void, true>::__call[abi:ue170006]<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__type_traits/invoke.h:415:5 (libyrpc.so+0x1befcb)
    #15 std::__function::__alloc_func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()[abi:ue170006]() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:192:16 (libyrpc.so+0x1befcb)
    #16 std::__function::__func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()() ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:363:12 (libyrpc.so+0x1befcb)
I1120 18:50:40.829716 196916 vector_lsm-test.cc:291] Search result: [{ distance: 0 base_table_key: 7665727465785F31 }, { distance: 1 base_table_key: 7665727465785F32 }, { distance: 1 base_table_key: 7665727465785F33 }, { distance: 1 base_table_key: 7665727465785F35 }, { distance: 1 base_table_key: 7665727465785F39 }]
    #17 std::__function::__value_func<void ()>::operator()[abi:ue170006]() const ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:517:16 (libyb_util.so+0x43615f)
    #18 std::function<void ()>::operator()() const ${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1168:12 (libyb_util.so+0x43615f)
    #19 yb::Thread::SuperviseThread(void*) ${YB_SRC_ROOT}/src/yb/util/thread.cc:888:3 (libyb_util.so+0x43615f)

Issue Type

kind/failing-test

Warning: Please confirm that this issue does not contain any sensitive information