search

pingcap / tiflash

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial
https://docs.pingcap.com/tidb/stable/tiflash-overview
Apache License 2.0
941 stars 409 forks source link

Failed test `StoreIngestTest.ConcurrentIngestAndWrite` under tsan #9257

Closed JaySon-Huang closed 1 month ago

JaySon-Huang commented 1 month ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. Build gtests_dbms under tsan build type
  2. Run gtests_dbms --gtest_filter=*StoreIngestTest.ConcurrentIngestAndWrite*

2. What did you expect to see? (Required)

3. What did you see instead (Required)

test failed or get blocked

4. What is your TiFlash version? (Required)

master

JaySon-Huang commented 1 month ago 
ThreadSanitizer: CHECK failed: sanitizer_deadlock_detector.h:67 "((n_all_locks_)) < (((sizeof(all_locks_with_contexts_)/sizeof((all_locks_with_contexts_)[0]))))" (0x40, 0x40) (tid=1691207)

The main thread get blocked in, we need to figure out the other thread cause potential "dead lock"

image

JaySon-Huang commented 1 month ago

The TSAN will failed when one thread hold locks larger than 64. This tests will try to do "segmentMerge" on many segments (more than 500) so causing this problem.

The code in question is in lib/sanitizer_common/sanitizer_deadlock_detector.h It limits the number of simultaneously held locks in a given thread to an arbitrary large number 64. If you hold 65 locks in one thread at once, this will fail.

https://github.com/google/sanitizers/issues/950

JaySon-Huang commented 1 month ago

another data race reported on xxh3, I'll suppress it.

==================
WARNING: ThreadSanitizer: data race (pid=2253513)
  Read of size 8 at 0x000014d2ce48 by thread T154:
    #0 XXH3_hashLong_64b_withSeed_selection /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:665:20 (gtests_dbms+0x103e7d99)
    #1 XXH3_64bits_internal /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/cmake_unofficial/../xxhash.h:3860:12 (gtests_dbms+0x103e7d99)
    #2 XXH3_64bits_withSeed_dispatch /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:671:12 (gtests_dbms+0x103e7d99)
    #3 DB::Digest::XXH3::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:126:17 (gtests_dbms+0xc0b4848)
    #4 DB::UnifiedDigest<DB::Digest::XXH3>::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:195:70 (gtests_dbms+0xc0b4848)
    #5 DB::DM::DMFileMetaV2::finalize(DB::WriteBuffer&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::WriteLimiter> const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileMetaV2.cpp:202:17 (gtests_dbms+0xc2acadf)
    #6 DB::DM::DMFileWriter::finalizeMeta() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:178:19 (gtests_dbms+0xc2e790d)
    #7 DB::DM::DMFileWriter::finalize() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:173:5 (gtests_dbms+0xc2e790d)
    #8 DB::DM::DMFileBlockOutputStream::writeSuffix() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h:44:33 (gtests_dbms+0xc181c24)
    #9 DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:205:20 (gtests_dbms+0xc181c24)
    #10 DB::DM::tests::genDMFile(std::__1::shared_ptr<DB::DM::DeltaMergeStore>, DB::DM::DMContext&, DB::Block const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:290:19 (gtests_dbms+0x58f8dc7)
    #11 DB::DM::tests::SimplePKTestBasic::ingestFiles(DB::DM::tests::SimplePKTestBasic::IngestFilesOptions const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:323:18 (gtests_dbms+0x58f95c3)
    #12 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:229:17 (gtests_dbms+0x57fbe1b)
    #13 decltype(static_cast<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(fp)()) std::__1::__invoke<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x57fbe1b)
    #14 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x57fbe1b)
    #15 std::__1::__function::__alloc_func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x57fbe1b)
    #16 std::__1::__function::__func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x57fbe1b)
    #17 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd81811)
    #18 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd81811)
    #19 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd81811)
    #20 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd849fc)
    #21 decltype(static_cast<void>(fp)()) std::__1::__invoke_constexpr<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3924:1 (gtests_dbms+0xfd849fc)
    #22 decltype(auto) std::__1::__apply_tuple_impl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1536:1 (gtests_dbms+0xfd849fc)
    #23 decltype(auto) std::__1::apply<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1545:1 (gtests_dbms+0xfd849fc)
    #24 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0xfd849fc)
    #25 decltype(static_cast<void>(fp)()) std::__1::__invoke<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd849fc)
    #26 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xfd849fc)
    #27 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xfd849fc)
    #28 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xfd849fc)
    #29 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd7f05a)
    #30 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd7f05a)
    #31 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd7f05a)
    #32 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd82ab0)
    #33 decltype(static_cast<void>(fp)()) std::__1::__invoke<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd82ab0)
    #34 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xfd82ab0)
    #35 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xfd82ab0)

  Previous write of size 8 at 0x000014d2ce48 by thread T157:
    #0 memcpy /root/llvm-project/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:827:5 (gtests_dbms+0x2c0a7f0)
    #1 setDispatch /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:639:18 (gtests_dbms+0x103e7e38)
    #2 XXH3_hashLong_64b_withSeed_selection /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:665:45 (gtests_dbms+0x103e7e38)
    #3 XXH3_64bits_internal /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/cmake_unofficial/../xxhash.h:3860:12 (gtests_dbms+0x103e7e38)
    #4 XXH3_64bits_withSeed_dispatch /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:671:12 (gtests_dbms+0x103e7e38)
    #5 DB::Digest::XXH3::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:126:17 (gtests_dbms+0xc0b4848)
    #6 DB::UnifiedDigest<DB::Digest::XXH3>::update(void const*, unsigned long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/Checksum.h:195:70 (gtests_dbms+0xc0b4848)
    #7 DB::DM::DMFileMetaV2::finalize(DB::WriteBuffer&, std::__1::shared_ptr<DB::FileProvider> const&, std::__1::shared_ptr<DB::WriteLimiter> const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileMetaV2.cpp:202:17 (gtests_dbms+0xc2acadf)
    #8 DB::DM::DMFileWriter::finalizeMeta() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:178:19 (gtests_dbms+0xc2e790d)
    #9 DB::DM::DMFileWriter::finalize() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileWriter.cpp:173:5 (gtests_dbms+0xc2e790d)
    #10 DB::DM::DMFileBlockOutputStream::writeSuffix() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/File/DMFileBlockOutputStream.h:44:33 (gtests_dbms+0xc181c24)
    #11 DB::DM::writeIntoNewDMFile(DB::DM::DMContext&, std::__1::shared_ptr<std::__1::vector<DB::DM::ColumnDefine, std::__1::allocator<DB::DM::ColumnDefine> > > const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/Segment.cpp:205:20 (gtests_dbms+0xc181c24)
    #12 DB::DM::tests::genDMFile(std::__1::shared_ptr<DB::DM::DeltaMergeStore>, DB::DM::DMContext&, DB::Block const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:290:19 (gtests_dbms+0x58f8dc7)
    #13 DB::DM::tests::SimplePKTestBasic::ingestFiles(DB::DM::tests::SimplePKTestBasic::IngestFilesOptions const&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_simple_pk_test_basic.cpp:323:18 (gtests_dbms+0x58f95c3)
    #14 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:229:17 (gtests_dbms+0x57fbe1b)
    #15 decltype(static_cast<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(fp)()) std::__1::__invoke<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0x57fbe1b)
    #16 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&>(DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0x57fbe1b)
    #17 std::__1::__function::__alloc_func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0x57fbe1b)
    #18 std::__1::__function::__func<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0, std::__1::allocator<DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody()::$_0>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0x57fbe1b)
    #19 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd81811)
    #20 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd81811)
    #21 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::worker(std::__1::__list_iterator<DB::ThreadFromGlobalPoolImpl<false>, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd81811)
    #22 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd849fc)
    #23 decltype(static_cast<void>(fp)()) std::__1::__invoke_constexpr<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3924:1 (gtests_dbms+0xfd849fc)
    #24 decltype(auto) std::__1::__apply_tuple_impl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1536:1 (gtests_dbms+0xfd849fc)
    #25 decltype(auto) std::__1::apply<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/tuple:1545:1 (gtests_dbms+0xfd849fc)
    #26 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()::operator()() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:236:17 (gtests_dbms+0xfd849fc)
    #27 decltype(static_cast<void>(fp)()) std::__1::__invoke<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd849fc)
    #28 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&>(DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/invoke.h:61:9 (gtests_dbms+0xfd849fc)
    #29 std::__1::__function::__alloc_func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:171:16 (gtests_dbms+0xfd849fc)
    #30 std::__1::__function::__func<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), std::__1::allocator<DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'()>, void ()>::operator()() /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:345:12 (gtests_dbms+0xfd849fc)
    #31 std::__1::__function::__value_func<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:498:16 (gtests_dbms+0xfd7f05a)
    #32 std::__1::function<void ()>::operator()() const /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__functional/function.h:1175:12 (gtests_dbms+0xfd7f05a)
    #33 DB::ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:306:17 (gtests_dbms+0xfd7f05a)
    #34 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()::operator()() const /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:73 (gtests_dbms+0xfd82ab0)
    #35 decltype(static_cast<void>(fp)()) std::__1::__invoke<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/type_traits:3918:1 (gtests_dbms+0xfd82ab0)
    #36 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(std::__1::tuple<void, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>&, std::__1::__tuple_indices<>) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:280:5 (gtests_dbms+0xfd82ab0)
    #37 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()> >(void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:291:5 (gtests_dbms+0xfd82ab0)

  Location is global 'g_dispatch' of size 32 at 0x000014d2ce40 (gtests_dbms+0x000014d2ce48)

  Thread T154 'ThreadPool' (tid=2253846, running) created by main thread at:
    #0 pthread_create /root/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2c00bcd)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xfd824e8)
    #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xfd824e8)
    #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7d67c)
    #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0xfd841c4)
    #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0xfd841c4)
    #6 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7fb1b)
    #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleOrThrowOnError(std::__1::function<void ()>, long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:182:5 (gtests_dbms+0xfd7f712)
    #8 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:224:15 (gtests_dbms+0x57f8dc3)
    #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc603115)
    #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc603115)
    #11 testing::Test::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc5dccd5)
    #12 testing::TestInfo::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc5de504)
    #13 testing::TestCase::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc5df1cd)
    #14 testing::internal::UnitTestImpl::RunAllTests() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc5ef380)
    #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc6045b5)
    #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc6045b5)
    #17 testing::UnitTest::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc5ee668)
    #18 RUN_ALL_TESTS() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x613bd2e)
    #19 main /data/nvme0n1/jaysonhuang/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x613bd2e)

  Thread T157 'ThreadPool' (tid=2253849, running) created by main thread at:
    #0 pthread_create /root/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:977:3 (gtests_dbms+0x2c00bcd)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/__threading_support:514:10 (gtests_dbms+0xfd824e8)
    #2 std::__1::thread::thread<void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'(), void>(void&&) /data/nvme0n1/ra_common/tiflash-env-13/sysroot/bin/../include/c++/v1/thread:307:16 (gtests_dbms+0xfd824e8)
    #3 void DB::ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7d67c)
    #4 DB::ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, long, unsigned long, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:198:5 (gtests_dbms+0xfd841c4)
    #5 DB::ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>(void&&) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.h:219:38 (gtests_dbms+0xfd841c4)
    #6 void DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:160:35 (gtests_dbms+0xfd7fb1b)
    #7 DB::ThreadPoolImpl<DB::ThreadFromGlobalPoolImpl<false> >::scheduleOrThrowOnError(std::__1::function<void ()>, long) /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Common/UniThreadPool.cpp:182:5 (gtests_dbms+0xfd7f712)
    #8 DB::DM::tests::StoreIngestTest_ConcurrentIngestAndWrite_Test::TestBody() /data/nvme0n1/jaysonhuang/tiflash/dbms/src/Storages/DeltaMerge/tests/gtest_dm_ingest.cpp:224:15 (gtests_dbms+0x57f8dc3)
    #9 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc603115)
    #10 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc603115)
    #11 testing::Test::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2517:5 (gtests_dbms+0xc5dccd5)
    #12 testing::TestInfo::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2693:11 (gtests_dbms+0xc5de504)
    #13 testing::TestCase::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2811:28 (gtests_dbms+0xc5df1cd)
    #14 testing::internal::UnitTestImpl::RunAllTests() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:5177:43 (gtests_dbms+0xc5ef380)
    #15 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2443:10 (gtests_dbms+0xc6045b5)
    #16 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:2479:14 (gtests_dbms+0xc6045b5)
    #17 testing::UnitTest::Run() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/src/gtest.cc:4786:10 (gtests_dbms+0xc5ee668)
    #18 RUN_ALL_TESTS() /data/nvme0n1/jaysonhuang/tiflash/contrib/googletest/googletest/include/gtest/gtest.h:2341:46 (gtests_dbms+0x613bd2e)
    #19 main /data/nvme0n1/jaysonhuang/tiflash/dbms/src/TestUtils/gtests_dbms_main.cpp:116:16 (gtests_dbms+0x613bd2e)

SUMMARY: ThreadSanitizer: data race /data/nvme0n1/jaysonhuang/tiflash/contrib/xxHash/xxh_x86dispatch.c:665:20 in XXH3_hashLong_64b_withSeed_selection
==================