search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.65k stars 3.26k forks source link

[Bug] dangling pointer in scanner context #18700

Closed TangSiyang2001 closed 1 year ago

TangSiyang2001 commented 1 year ago

Search before asking

Version

current master

What's Wrong?

When running p0 in CI, found BE crash, and it happended 3 times in 4 runnings total, which may thought to be a problem.

According to be.out, a dangling pointer may happen in ScannerContext::clear_and_join, related to ScannerContext::return_free_block.

be.out

start time: Sat 15 Apr 2023 11:40:29 PM CST
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0415 23:40:29.853989 1715262 doris_main.cpp:322] enable_fuzzy_mode is true, set fuzzy configs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
=================================================================
==1715262==ERROR: AddressSanitizer: heap-use-after-free on address 0x6030034fe5e8 at pc 0x55dacf08b288 bp 0x7f784cf078b0 sp 0x7f784cf078a0
READ of size 8 at 0x6030034fe5e8 thread T278 (TaskSchedulerTh)
    #0 0x55dacf08b287 in std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >::~unique_ptr() /var/local/ldb-toolchain/include/c++/11/bits/unique_ptr.h:360
    #1 0x55dacf08b287 in void std::_Destroy<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*) /var/local/ldb-toolchain/include/c++/11/bits/stl_construct.h:140
    #2 0x55dacf08b287 in void std::_Destroy_aux<false>::__destroy<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*>(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*) /var/local/ldb-toolchain/include/c++/11/bits/stl_construct.h:152
    #3 0x55dacf08b287 in void std::_Destroy<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*>(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*) /var/local/ldb-toolchain/include/c++/11/bits/stl_construct.h:185
    #4 0x55dacf08b287 in void std::_Destroy<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >&) /var/local/ldb-toolchain/include/c++/11/bits/alloc_traits.h:746
    #5 0x55dacf08b287 in std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::_M_erase_at_end(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*) /var/local/ldb-toolchain/include/c++/11/bits/stl_vector.h:1796
    #6 0x55dacf08b287 in std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::clear() /var/local/ldb-toolchain/include/c++/11/bits/stl_vector.h:1499
    #7 0x55dacf08b287 in doris::vectorized::ScannerContext::clear_and_join(doris::vectorized::VScanNode*, doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/scanner_context.cpp:275
    #8 0x55dacf100dc3 in doris::vectorized::VScanNode::release_resource(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/vscan_node.cpp:424
    #9 0x55dacf7da07d in doris::pipeline::StreamingOperator<doris::pipeline::ScanOperatorBuilder>::close(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/operator.h:349
    #10 0x55dacf7ac50f in doris::pipeline::PipelineTask::close() /root/doris/be/src/pipeline/pipeline_task.cpp:231
    #11 0x55dacf7c0343 in doris::pipeline::TaskScheduler::_try_close_task(doris::pipeline::PipelineTask*, doris::pipeline::PipelineTaskState) /root/doris/be/src/pipeline/task_scheduler.cpp:302
    #12 0x55dacf7ccf1f in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:272
    #13 0x55dacf7cd84d in void std::__invoke_impl<void, void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>(std::__invoke_memfun_deref, void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #14 0x55dacf7cd84d in std::__invoke_result<void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>::type std::__invoke<void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>(void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #15 0x55dacf7cd84d in void std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #16 0x55dacf7cd84d in void std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #17 0x55dacf7cd84d in void std::__invoke_impl<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>(std::__invoke_other, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #18 0x55dacf7cd84d in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>(std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #19 0x55dacf7cd84d in std::_Function_handler<void (), std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #20 0x55dab81a6070 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #21 0x55dab81a6070 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:46
    #22 0x55dab81a3f88 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:529
    #23 0x55dab81a5c25 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #24 0x55dab81a5c25 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #25 0x55dab81a5c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #26 0x55dab81a5c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #27 0x55dab81a5c25 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #28 0x55dab81a5c25 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #29 0x55dab81a5c25 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #30 0x55dab817ea83 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #31 0x55dab817ea83 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:453
    #32 0x7f7967ed5608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #33 0x7f7967cab132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

0x6030034fe5e8 is located 8 bytes inside of 32-byte region [0x6030034fe5e0,0x6030034fe600)
freed by thread T267 (TaskSchedulerTh) here:
    #0 0x55dab54ee767 in operator delete(void*, unsigned long) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb47a767)
    #1 0x55dab6e4664e in __gnu_cxx::new_allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >::deallocate(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, unsigned long) /var/local/ldb-toolchain/include/c++/11/ext/new_allocator.h:139
    #2 0x55dab6e4664e in std::allocator_traits<std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::deallocate(std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >&, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, unsigned long) /var/local/ldb-toolchain/include/c++/11/bits/alloc_traits.h:492
    #3 0x55dab6e4664e in std::_Vector_base<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::_M_deallocate(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, unsigned long) /var/local/ldb-toolchain/include/c++/11/bits/stl_vector.h:354
    #4 0x55dab6e4664e in void std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::_M_realloc_insert<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >(__gnu_cxx::__normal_iterator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > > >, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >&&) /var/local/ldb-toolchain/include/c++/11/bits/vector.tcc:500
    #5 0x55dacf0925f4 in std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >& std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::emplace_back<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >&&) /var/local/ldb-toolchain/include/c++/11/bits/vector.tcc:121
    #6 0x55dacf07eb55 in doris::vectorized::ScannerContext::return_free_block(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >) /root/doris/be/src/vec/exec/scan/scanner_context.cpp:141
    #7 0x55dacf0b8a76 in doris::vectorized::VScanNode::get_next(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/vec/exec/scan/vscan_node.cpp:220
    #8 0x55dab7a99cac in doris::ExecNode::pull(doris::RuntimeState*, doris::vectorized::Block*, bool*) /root/doris/be/src/exec/exec_node.h:128
    #9 0x55dacf7d9b4d in doris::Status std::__invoke_impl<doris::Status, doris::Status (doris::ExecNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::__invoke_memfun_deref, doris::Status (doris::ExecNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #10 0x55dacf7d9b4d in std::__invoke_result<doris::Status (doris::ExecNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*, doris::vectorized::Block*, bool*>::type std::__invoke<doris::Status (doris::ExecNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(doris::Status (doris::ExecNode::*&)(doris::RuntimeState*, doris::vectorized::Block*, bool*), doris::vectorized::VScanNode*&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:97
    #11 0x55dacf7d9b4d in doris::Status std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::__call<doris::Status, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&, 0ul, 1ul, 2ul, 3ul>(std::tuple<doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) /var/local/ldb-toolchain/include/c++/11/functional:422
    #12 0x55dacf7d9b4d in doris::Status std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>::operator()<doris::RuntimeState*, doris::vectorized::Block*, bool*, doris::Status>(doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb-toolchain/include/c++/11/functional:505
    #13 0x55dacf7d9b4d in doris::Status std::__invoke_impl<doris::Status, std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::__invoke_other, std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #14 0x55dacf7d9b4d in std::enable_if<is_invocable_r_v<doris::Status, std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>, doris::Status>::type std::__invoke_r<doris::Status, std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*, doris::vectorized::Block*, bool*>(std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)>&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:116
    #15 0x55dacf7d9b4d in std::_Function_handler<doris::Status (doris::RuntimeState*, doris::vectorized::Block*, bool*), std::_Bind<doris::Status (doris::ExecNode::*(doris::vectorized::VScanNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)> >::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:292
    #16 0x55dab7a9422d in std::function<doris::Status (doris::RuntimeState*, doris::vectorized::Block*, bool*)>::operator()(doris::RuntimeState*, doris::vectorized::Block*, bool*) const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #17 0x55dab7a9422d in doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*, std::function<doris::Status (doris::RuntimeState*, doris::vectorized::Block*, bool*)> const&, bool) /root/doris/be/src/exec/exec_node.cpp:586
    #18 0x55dacf7df034 in doris::pipeline::SourceOperator<doris::pipeline::ScanOperatorBuilder>::get_block(doris::RuntimeState*, doris::vectorized::Block*, doris::pipeline::SourceState&) /root/doris/be/src/pipeline/exec/operator.h:400
    #19 0x55dacf7a8aaf in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:188
    #20 0x55dacf7cb3e6 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:249
    #21 0x55dacf7cd84d in void std::__invoke_impl<void, void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>(std::__invoke_memfun_deref, void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #22 0x55dacf7cd84d in std::__invoke_result<void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>::type std::__invoke<void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>(void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #23 0x55dacf7cd84d in void std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #24 0x55dacf7cd84d in void std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #25 0x55dacf7cd84d in void std::__invoke_impl<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>(std::__invoke_other, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #26 0x55dacf7cd84d in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>(std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #27 0x55dacf7cd84d in std::_Function_handler<void (), std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #28 0x55dab81a6070 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #29 0x55dab81a6070 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:46
    #30 0x55dab81a3f88 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:529
    #31 0x55dab81a5c25 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #32 0x55dab81a5c25 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #33 0x55dab81a5c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #34 0x55dab81a5c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #35 0x55dab81a5c25 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #36 0x55dab81a5c25 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #37 0x55dab81a5c25 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #38 0x55dab817ea83 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #39 0x55dab817ea83 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:453
    #40 0x7f7967ed5608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477

previously allocated by thread T281 (TaskSchedulerTh) here:
    #0 0x55dab54ed707 in operator new(unsigned long) (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb479707)
    #1 0x55dab6e46521 in __gnu_cxx::new_allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >::allocate(unsigned long, void const*) /var/local/ldb-toolchain/include/c++/11/ext/new_allocator.h:121
    #2 0x55dab6e46521 in std::allocator_traits<std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::allocate(std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >&, unsigned long) /var/local/ldb-toolchain/include/c++/11/bits/alloc_traits.h:460
    #3 0x55dab6e46521 in std::_Vector_base<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::_M_allocate(unsigned long) /var/local/ldb-toolchain/include/c++/11/bits/stl_vector.h:346
    #4 0x55dab6e46521 in void std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::_M_realloc_insert<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >(__gnu_cxx::__normal_iterator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >*, std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > > >, std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >&&) /var/local/ldb-toolchain/include/c++/11/bits/vector.tcc:440
    #5 0x55dacf0925f4 in std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >& std::vector<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >, std::allocator<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > > >::emplace_back<std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> > >(std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >&&) /var/local/ldb-toolchain/include/c++/11/bits/vector.tcc:121
    #6 0x55dacf081b19 in doris::vectorized::ScannerContext::init() /root/doris/be/src/vec/exec/scan/scanner_context.cpp:94
    #7 0x55dacf0a0c48 in doris::vectorized::VScanNode::_start_scanners(std::__cxx11::list<doris::vectorized::VScanner*, std::allocator<doris::vectorized::VScanner*> > const&) /root/doris/be/src/vec/exec/scan/vscan_node.cpp:280
    #8 0x55dacf0a3950 in doris::vectorized::VScanNode::_prepare_scanners() /root/doris/be/src/vec/exec/scan/vscan_node.cpp:1372
    #9 0x55dacf0fc8b8 in doris::vectorized::VScanNode::alloc_resource(doris::RuntimeState*) /root/doris/be/src/vec/exec/scan/vscan_node.cpp:146
    #10 0x55dacf7de226 in doris::pipeline::StreamingOperator<doris::pipeline::ScanOperatorBuilder>::open(doris::RuntimeState*) /root/doris/be/src/pipeline/exec/operator.h:333
    #11 0x55dacf7997fa in doris::pipeline::PipelineTask::_open() /root/doris/be/src/pipeline/pipeline_task.cpp:115
    #12 0x55dacf7a7612 in doris::pipeline::PipelineTask::execute(bool*) /root/doris/be/src/pipeline/pipeline_task.cpp:144
    #13 0x55dacf7cb3e6 in doris::pipeline::TaskScheduler::_do_work(unsigned long) /root/doris/be/src/pipeline/task_scheduler.cpp:249
    #14 0x55dacf7cd84d in void std::__invoke_impl<void, void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>(std::__invoke_memfun_deref, void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #15 0x55dacf7cd84d in std::__invoke_result<void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>::type std::__invoke<void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&>(void (doris::pipeline::TaskScheduler::*&)(unsigned long), doris::pipeline::TaskScheduler*&, unsigned long&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #16 0x55dacf7cd84d in void std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>::__call<void, , 0ul, 1ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #17 0x55dacf7cd84d in void std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #18 0x55dacf7cd84d in void std::__invoke_impl<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>(std::__invoke_other, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #19 0x55dacf7cd84d in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&>(std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #20 0x55dacf7cd84d in std::_Function_handler<void (), std::_Bind<void (doris::pipeline::TaskScheduler::*(doris::pipeline::TaskScheduler*, unsigned long))(unsigned long)> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #21 0x55dab81a6070 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #22 0x55dab81a6070 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:46
    #23 0x55dab81a3f88 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:529
    #24 0x55dab81a5c25 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #25 0x55dab81a5c25 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #26 0x55dab81a5c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #27 0x55dab81a5c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #28 0x55dab81a5c25 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #29 0x55dab81a5c25 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #30 0x55dab81a5c25 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #31 0x55dab817ea83 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #32 0x55dab817ea83 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:453
    #33 0x7f7967ed5608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477

Thread T278 (TaskSchedulerTh) created by T0 here:
    #0 0x55dab5490061 in __interceptor_pthread_create (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb41c061)
    #1 0x55dab817aac9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.cpp:407
    #2 0x55dab8191fce in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.h:57
    #3 0x55dab8191fce in doris::ThreadPool::create_thread() /root/doris/be/src/util/threadpool.cpp:598
    #4 0x55dab819ba1d in doris::ThreadPool::init() /root/doris/be/src/util/threadpool.cpp:257
    #5 0x55dacf7c283c in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool> >*) const /root/doris/be/src/util/threadpool.h:114
    #6 0x55dacf7c283c in doris::pipeline::TaskScheduler::start() /root/doris/be/src/pipeline/task_scheduler.cpp:200
    #7 0x55dab786e5f5 in doris::ExecEnv::init_pipeline_task_scheduler() /root/doris/be/src/runtime/exec_env_init.cpp:173
    #8 0x55dab788425c in doris::ExecEnv::_init(std::vector<doris::StorePath, std::allocator<doris::StorePath> > const&) /root/doris/be/src/runtime/exec_env_init.cpp:123
    #9 0x55dab7886560 in doris::ExecEnv::init(doris::ExecEnv*, std::vector<doris::StorePath, std::allocator<doris::StorePath> > const&) /root/doris/be/src/runtime/exec_env_init.cpp:73
    #10 0x55dab553d284 in main /root/doris/be/src/service/doris_main.cpp:436
    #11 0x7f7967bb0082 in __libc_start_main ../csu/libc-start.c:308

Thread T267 (TaskSchedulerTh) created by T0 here:
    #0 0x55dab5490061 in __interceptor_pthread_create (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb41c061)
    #1 0x55dab817aac9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.cpp:407
    #2 0x55dab8191fce in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.h:57
    #3 0x55dab8191fce in doris::ThreadPool::create_thread() /root/doris/be/src/util/threadpool.cpp:598
    #4 0x55dab819ba1d in doris::ThreadPool::init() /root/doris/be/src/util/threadpool.cpp:257
    #5 0x55dacf7c283c in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool> >*) const /root/doris/be/src/util/threadpool.h:114
    #6 0x55dacf7c283c in doris::pipeline::TaskScheduler::start() /root/doris/be/src/pipeline/task_scheduler.cpp:200
    #7 0x55dab786e5f5 in doris::ExecEnv::init_pipeline_task_scheduler() /root/doris/be/src/runtime/exec_env_init.cpp:173
    #8 0x55dab788425c in doris::ExecEnv::_init(std::vector<doris::StorePath, std::allocator<doris::StorePath> > const&) /root/doris/be/src/runtime/exec_env_init.cpp:123
    #9 0x55dab7886560 in doris::ExecEnv::init(doris::ExecEnv*, std::vector<doris::StorePath, std::allocator<doris::StorePath> > const&) /root/doris/be/src/runtime/exec_env_init.cpp:73
    #10 0x55dab553d284 in main /root/doris/be/src/service/doris_main.cpp:436
    #11 0x7f7967bb0082 in __libc_start_main ../csu/libc-start.c:308

Thread T281 (TaskSchedulerTh) created by T0 here:
    #0 0x55dab5490061 in __interceptor_pthread_create (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb41c061)
    #1 0x55dab817aac9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.cpp:407
    #2 0x55dab8191fce in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.h:57
    #3 0x55dab8191fce in doris::ThreadPool::create_thread() /root/doris/be/src/util/threadpool.cpp:598
    #4 0x55dab819ba1d in doris::ThreadPool::init() /root/doris/be/src/util/threadpool.cpp:257
    #5 0x55dacf7c283c in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool> >*) const /root/doris/be/src/util/threadpool.h:114
    #6 0x55dacf7c283c in doris::pipeline::TaskScheduler::start() /root/doris/be/src/pipeline/task_scheduler.cpp:200
    #7 0x55dab786e5f5 in doris::ExecEnv::init_pipeline_task_scheduler() /root/doris/be/src/runtime/exec_env_init.cpp:173
    #8 0x55dab788425c in doris::ExecEnv::_init(std::vector<doris::StorePath, std::allocator<doris::StorePath> > const&) /root/doris/be/src/runtime/exec_env_init.cpp:123
    #9 0x55dab7886560 in doris::ExecEnv::init(doris::ExecEnv*, std::vector<doris::StorePath, std::allocator<doris::StorePath> > const&) /root/doris/be/src/runtime/exec_env_init.cpp:73
    #10 0x55dab553d284 in main /root/doris/be/src/service/doris_main.cpp:436
    #11 0x7f7967bb0082 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-use-after-free /var/local/ldb-toolchain/include/c++/11/bits/unique_ptr.h:360 in std::unique_ptr<doris::vectorized::Block, std::default_delete<doris::vectorized::Block> >::~unique_ptr()
Shadow bytes around the buggy address:
  0x0c0680697c60: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd
  0x0c0680697c70: fd fd fa fa 00 00 00 00 fa fa fd fd fd fa fa fa
  0x0c0680697c80: fa fa fa fa fa fa fa fa fa fa fa fa fd fd fd fa
  0x0c0680697c90: fa fa fd fd fd fa fa fa fa fa fa fa fa fa 00 00
  0x0c0680697ca0: 00 fa fa fa fa fa fa fa fa fa fd fd fd fa fa fa
=>0x0c0680697cb0: 00 00 00 fa fa fa fd fd fd fd fa fa fd[fd]fd fd
  0x0c0680697cc0: fa fa 00 00 00 fa fa fa 00 00 00 00 fa fa 00 00
  0x0c0680697cd0: 00 fa fa fa fd fd fd fa fa fa fd fd fd fd fa fa
  0x0c0680697ce0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c0680697cf0: fa fa fa fa fa fa fa fa 00 00 00 00 fa fa 00 00
  0x0c0680697d00: 00 fa fa fa fa fa fa fa fa fa 00 00 00 fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1715262==ABORTING
start time: Sun 16 Apr 2023 12:00:42 AM CST
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0416 00:00:42.706631 1730339 doris_main.cpp:322] enable_fuzzy_mode is true, set fuzzy configs
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory]
=================================================================
==1730339==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f690a340c48 at pc 0x55fb75ff8b17 bp 0x7f690a340c10 sp 0x7f690a3403b8
WRITE of size 24 at 0x7f690a340c48 thread T792 (TaskWorkerPool.)
    #0 0x55fb75ff8b16 in __interceptor_sigaltstack.part.0 (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb417b16)
    #1 0x55fb7605e8bf in __asan::PlatformUnpoisonStacks() (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb47d8bf)
    #2 0x55fb76064224 in __asan_handle_no_return (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb483224)
    #3 0x55fb793f6d02 in apache::thrift::protocol::TProtocol::decrementOutputRecursionDepth() /var/local/thirdparty/installed/include/thrift/protocol/TProtocol.h:541
    #4 0x55fb793f6d02 in apache::thrift::protocol::TOutputRecursionTracker::~TOutputRecursionTracker() /var/local/thirdparty/installed/include/thrift/protocol/TProtocol.h:636
    #5 0x55fb793f6d02 in doris::TTabletInfo::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/MasterService_types.cpp:465
    #6 0x55fb793efdc4 in doris::TTablet::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/MasterService_types.cpp:1193
    #7 0x55fb793f7e64 in doris::TReportRequest::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/MasterService_types.cpp:1908
    #8 0x55fb791538a7 in doris::FrontendService_report_pargs::write(apache::thrift::protocol::TProtocol*) const /root/doris/gensrc/build/gen_cpp/FrontendService.cpp:1391
    #9 0x55fb79185830 in doris::FrontendServiceClient::send_report(doris::TReportRequest const&) /root/doris/gensrc/build/gen_cpp/FrontendService.cpp:5998
    #10 0x55fb791fd2ec in doris::FrontendServiceClient::report(doris::TMasterResult&, doris::TReportRequest const&) /root/doris/gensrc/build/gen_cpp/FrontendService.cpp:5987
    #11 0x55fb7785473c in doris::MasterServerClient::report(doris::TReportRequest const&, doris::TMasterResult*) /root/doris/be/src/agent/utils.cpp:108
    #12 0x55fb777ee004 in doris::TaskWorkerPool::_handle_report(doris::TReportRequest const&, doris::TaskWorkerPool::ReportType) /root/doris/be/src/agent/task_worker_pool.cpp:1667
    #13 0x55fb777f4d85 in doris::TaskWorkerPool::_report_tablet_worker_thread_callback() /root/doris/be/src/agent/task_worker_pool.cpp:1378
    #14 0x55fb778345d1 in void std::__invoke_impl<void, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&>(std::__invoke_memfun_deref, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #15 0x55fb778345d1 in std::enable_if<is_invocable_r_v<void, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&>, void>::type std::__invoke_r<void, void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&>(void (doris::TaskWorkerPool::*&)(), doris::TaskWorkerPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #16 0x55fb778345d1 in void std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:570
    #17 0x55fb778345d1 in void std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>::operator()<>() /var/local/ldb-toolchain/include/c++/11/functional:629
    #18 0x55fb778345d1 in void std::__invoke_impl<void, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&>(std::__invoke_other, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #19 0x55fb778345d1 in std::enable_if<is_invocable_r_v<void, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&>(std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #20 0x55fb778345d1 in std::_Function_handler<void (), std::_Bind_result<void, void (doris::TaskWorkerPool::*(doris::TaskWorkerPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #21 0x55fb78d13070 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #22 0x55fb78d13070 in doris::FunctionRunnable::run() /root/doris/be/src/util/threadpool.cpp:46
    #23 0x55fb78d10f88 in doris::ThreadPool::dispatch_thread() /root/doris/be/src/util/threadpool.cpp:529
    #24 0x55fb78d12c25 in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:74
    #25 0x55fb78d12c25 in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:96
    #26 0x55fb78d12c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb-toolchain/include/c++/11/functional:420
    #27 0x55fb78d12c25 in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /var/local/ldb-toolchain/include/c++/11/functional:503
    #28 0x55fb78d12c25 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:61
    #29 0x55fb78d12c25 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&) /var/local/ldb-toolchain/include/c++/11/bits/invoke.h:111
    #30 0x55fb78d12c25 in std::_Function_handler<void (), std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()> >::_M_invoke(std::_Any_data const&) /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
    #31 0x55fb78ceba83 in std::function<void ()>::operator()() const /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:560
    #32 0x55fb78ceba83 in doris::Thread::supervise_thread(void*) /root/doris/be/src/util/thread.cpp:453
    #33 0x7f6b1006d608 in start_thread /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:477
    #34 0x7f6b0fe43132 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f132)

Address 0x7f690a340c48 is located in stack of thread T792 (TaskWorkerPool.) at offset 40 in frame
    #0 0x55fb778569bf in apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport, apache::thrift::protocol::TNetworkBigEndian>, apache::thrift::protocol::TProtocolDefaults>::writeI64_virt(long) /var/local/thirdparty/installed/include/thrift/protocol/TVirtualProtocol.h:380

  This frame has 1 object(s):
    [32, 40) 'net' <== Memory access at offset 40 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
Thread T792 (TaskWorkerPool.) created by T0 here:
    #0 0x55fb75ffd061 in __interceptor_pthread_create (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb41c061)
    #1 0x55fb78ce7ac9 in doris::Thread::start_thread(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void ()> const&, unsigned long, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.cpp:407
    #2 0x55fb78cfefce in doris::Status doris::Thread::create<void (doris::ThreadPool::*)(), doris::ThreadPool*>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, void (doris::ThreadPool::* const&)(), doris::ThreadPool* const&, scoped_refptr<doris::Thread>*) /root/doris/be/src/util/thread.h:57
    #3 0x55fb78cfefce in doris::ThreadPool::create_thread() /root/doris/be/src/util/threadpool.cpp:598
    #4 0x55fb78d08a1d in doris::ThreadPool::init() /root/doris/be/src/util/threadpool.cpp:257
    #5 0x55fb777df360 in doris::Status doris::ThreadPoolBuilder::build<doris::ThreadPool>(std::unique_ptr<doris::ThreadPool, std::default_delete<doris::ThreadPool> >*) const /root/doris/be/src/util/threadpool.h:114
    #6 0x55fb777df360 in doris::TaskWorkerPool::start() /root/doris/be/src/agent/task_worker_pool.cpp:222
    #7 0x55fb7890790a in doris::AgentServer::AgentServer(doris::ExecEnv*, doris::TMasterInfo const&) /root/doris/be/src/agent/agent_server.cpp:96
    #8 0x55fb788e1f2b in doris::BackendService::BackendService(doris::ExecEnv*) /root/doris/be/src/service/backend_service.cpp:68
    #9 0x55fb788ebfeb in doris::BackendService::create_service(doris::ExecEnv*, int, doris::ThriftServer**) /root/doris/be/src/service/backend_service.cpp:71
    #10 0x55fb760aa793 in main /root/doris/be/src/service/doris_main.cpp:462
    #11 0x7f6b0fd48082 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: stack-buffer-overflow (/mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster2/be/lib/doris_be+0xb417b16) in __interceptor_sigaltstack.part.0
Shadow bytes around the buggy address:
  0x0feda1460130: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0feda1460140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0feda1460150: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0feda1460160: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0feda1460170: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0feda1460180: 00 00 00 00 f1 f1 f1 f1 00[f3]f3 f3 00 00 00 00
  0x0feda1460190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0feda14601a0: f1 f1 f1 f1 f1 f1 01 f2 00 f2 f2 f2 00 f2 f2 f2
  0x0feda14601b0: 00 f2 f2 f2 00 f2 f2 f2 00 f3 f3 f3 00 00 00 00
  0x0feda14601c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0feda14601d0: f1 f1 f1 f1 f1 f1 01 f2 00 f2 f2 f2 00 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==1730339==ABORTING

What You Expected?

Run normally.

How to Reproduce?

Maybe run p0 several times.

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

TangSiyang2001 commented 1 year ago

resolved by PR#18701, data race at _free_blocks.