search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.65k stars 3.26k forks source link

[Bug] vectorized load use after free on master #11115

Closed yixiutt closed 2 years ago

yixiutt commented 2 years ago

Search before asking

Version

master

What's Wrong?

0 0x0000558c91016080 in doris::signal::(anonymous namespace)::FailureSignalHandler (signal_number=11, signal_info=0x7f9121eba5b0,

ucontext=0x7f9121eba480) at /home/zcp/repo_center/doris_master/be/src/common/signal_handler.h:372

1

2 0x0000558c91016080 in doris::signal::(anonymous namespace)::FailureSignalHandler (signal_number=6, signal_info=0x7f9121ebac70,

ucontext=0x7f9121ebab40) at /home/zcp/repo_center/doris_master/be/src/common/signal_handler.h:372

3

4 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50

5 0x00007f92146b0859 in __GI_abort () at abort.c:79

6 0x0000558c90fec42e in __sanitizer::Abort() ()

7 0x0000558c90ff7d18 in __sanitizer::Die() ()

8 0x0000558c90fd7ae4 in __asan::ScopedInErrorReport::~ScopedInErrorReport() ()

9 0x0000558c90fd73a1 in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) ()

10 0x0000558c90fd8117 in __asan_report_load8 ()

11 0x0000558c93ecb11a in doris::vectorized::Value::get_value (this=0x7f90d1cbc048)

at /home/zcp/repo_center/doris_master/be/src/vec/aggregate_functions/aggregate_function_window.h:214

12 0x0000558c93f24b92 in doris::vectorized::LeadAndLagData<long, true, false, doris::vectorized::Value>::insert_result_into (

this=0x7f90d1cbc048, to=...) at /home/zcp/repo_center/doris_master/be/src/vec/aggregate_functions/aggregate_function_window.h:271

13 0x0000558c93f0915c in doris::vectorized::WindowFunctionData<doris::vectorized::WindowFunctionLastData<doris::vectorized::LeadAndLagData<long, true, false, doris::vectorized::Value> > >::insert_result_into (this=0x607000aa2ed0, place=0x7f90d1cbc048 "", to=...)

at /home/zcp/repo_center/doris_master/be/src/vec/aggregate_functions/aggregate_function_window.h:508

14 0x0000558c91ac79c8 in doris::MemTable::_collect_vskiplist_results (this=0x61700047b600)

at /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:315

15 0x0000558c91ab45b0 in doris::MemTable::_do_flush (this=0x61700047b600, duration_ns=@0x7f9121ebf090: 0)

at /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:394

16 0x0000558c91ab374e in doris::MemTable::flush (this=0x61700047b600) at /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:367

17 0x0000558c91aa6b8c in doris::FlushToken::_flush_memtable (this=0x606001931480, memtable=..., submit_task_time=1216724600827163)

at /home/zcp/repo_center/doris_master/be/src/olap/memtable_flush_executor.cpp:85

18 0x0000558c91aad534 in std::__invoke_impl<void, void (doris::FlushToken::&)(std::shared_ptr, long), doris::FlushToken&, std::shared_ptr&, long&> (__f=

What You Expected?

fix

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

dataroaring commented 2 years ago

`================================================================= ==1878282==ERROR: AddressSanitizer: heap-use-after-free on address 0x603008858500 at pc 0x558c93ecb11a bp 0x7f9121ebc100 sp 0x7f9121ebc0f0 READ of size 8 at 0x603008858500 thread T862 (MemTableFlushTh)

0 0x558c93ecb119 in doris::vectorized::Value::get_value() const /home/zcp/repo_center/doris_master/be/src/vec/aggregate_functions/aggregate_function_window.h:214

#1 0x558c93f24b91 in doris::vectorized::LeadAndLagData<long, true, false, doris::vectorized::Value>::insert_result_into(doris::vectorized::IColumn&) const /home/zcp/repo_center/doris_master/be/src/vec/aggregate_functions/aggregate_function_window.h:271
#2 0x558c93f0915b in doris::vectorized::WindowFunctionData<doris::vectorized::WindowFunctionLastData<doris::vectorized::LeadAndLagData<long, true, false, doris::vectorized::Value> > >::insert_result_into(char const*, doris::vectorized::IColumn&) const /home/zcp/repo_center/doris_master/be/src/vec/aggregate_functions/aggregate_function_window.h:508
#3 0x558c91ac79c7 in void doris::MemTable::_collect_vskiplist_results<true>() /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:315
#4 0x558c91ab45af in doris::MemTable::_do_flush(long&) /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:394
#5 0x558c91ab374d in doris::MemTable::flush() /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:367
#6 0x558c91aa6b8b in doris::FlushToken::_flush_memtable(std::shared_ptr<doris::MemTable>, long) /home/zcp/repo_center/doris_master/be/src/olap/memtable_flush_executor.cpp:85
#7 0x558c91aad533 in void std::__invoke_impl<void, void (doris::FlushToken::*&)(std::shared_ptr<doris::MemTable>, long), doris::FlushToken*&, std::shared_ptr<doris::MemTable>&, long&>(std::__invoke_memfun_deref, void (doris::FlushToken::*&)(std::shared_ptr<doris::MemTable>, long), doris::FlushToken*&, std::shared_ptr<doris::MemTable>&, long&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:74
#8 0x558c91aad152 in std::__invoke_result<void (doris::FlushToken::*&)(std::shared_ptr<doris::MemTable>, long), doris::FlushToken*&, std::shared_ptr<doris::MemTable>&, long&>::type std::__invoke<void (doris::FlushToken::*&)(std::shared_ptr<doris::MemTable>, long), doris::FlushToken*&, std::shared_ptr<doris::MemTable>&, long&>(void (doris::FlushToken::*&)(std::shared_ptr<doris::MemTable>, long), doris::FlushToken*&, std::shared_ptr<doris::MemTable>&, long&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:96
#9 0x558c91aace4f in void std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>::__call<void, , 0ul, 1ul, 2ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul>) /var/local/ldb_toolchain/include/c++/11/functional:420
#10 0x558c91aacb16 in void std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>::operator()<, void>() /var/local/ldb_toolchain/include/c++/11/functional:503
#11 0x558c91aac535 in void std::__invoke_impl<void, std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>&>(std::__invoke_other, std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
#12 0x558c91aabf99 in std::enable_if<is_invocable_r_v<void, std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>&>, void>::type std::__invoke_r<void, std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>&>(std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)>&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#13 0x558c91aaba18 in std::_Function_handler<void (), std::_Bind<void (doris::FlushToken::*(doris::FlushToken*, std::shared_ptr<doris::MemTable>, long))(std::shared_ptr<doris::MemTable>, long)> >::_M_invoke(std::_Any_data const&) /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#14 0x558c92876551 in std::function<void ()>::operator()() const /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#15 0x558c93096ebd in doris::FunctionRunnable::run() /home/zcp/repo_center/doris_master/be/src/util/threadpool.cpp:45
#16 0x558c930922e0 in doris::ThreadPool::dispatch_thread() /home/zcp/repo_center/doris_master/be/src/util/threadpool.cpp:540
#17 0x558c930b348d in void std::__invoke_impl<void, void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(std::__invoke_memfun_deref, void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:74
#18 0x558c930b2d2c in std::__invoke_result<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>::type std::__invoke<void (doris::ThreadPool::*&)(), doris::ThreadPool*&>(void (doris::ThreadPool::*&)(), doris::ThreadPool*&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:96
#19 0x558c930b20cb in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) /var/local/ldb_toolchain/include/c++/11/functional:420
#20 0x558c930b0bce in void std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>::operator()<, void>() /var/local/ldb_toolchain/include/c++/11/functional:503
#21 0x558c930ad735 in void std::__invoke_impl<void, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&>(std::__invoke_other, std::_Bind<void (doris::ThreadPool::*(doris::ThreadPool*))()>&log/be.out`
dataroaring commented 2 years ago

`0x603008858500 is located 0 bytes inside of 32-byte region [0x603008858500,0x603008858520) freed by thread T708 here:

0 0x558c90fd1767 in operator delete(void*, unsigned long) (/mnt/ssd01/doris-master/VEC_ASAN/be/lib/doris_be+0x918a767)

#1 0x558c9418bd9a in doris::vectorized::ColumnNullable::~ColumnNullable() /home/zcp/repo_center/doris_master/be/src/vec/columns/column_nullable.h:47
#2 0x558c911c2b44 in COW<doris::vectorized::IColumn>::release_ref() /home/zcp/repo_center/doris_master/be/src/vec/common/cow.h:99
#3 0x558c911b913d in COW<doris::vectorized::IColumn>::intrusive_ptr<doris::vectorized::IColumn const>::~intrusive_ptr() /home/zcp/repo_center/doris_master/be/src/vec/common/cow.h:133
#4 0x558c9117de1d in COW<doris::vectorized::IColumn>::immutable_ptr<doris::vectorized::IColumn>::~immutable_ptr() /home/zcp/repo_center/doris_master/be/src/vec/common/cow.h:255
#5 0x558c91637b7f in doris::vectorized::ColumnWithTypeAndName::~ColumnWithTypeAndName() /home/zcp/repo_center/doris_master/be/src/vec/core/column_with_type_and_name.h:35
#6 0x558c91637b9a in void std::_Destroy<doris::vectorized::ColumnWithTypeAndName>(doris::vectorized::ColumnWithTypeAndName*) /var/local/ldb_toolchain/include/c++/11/bits/stl_construct.h:140
#7 0x558c91636fbc in void std::_Destroy_aux<false>::__destroy<doris::vectorized::ColumnWithTypeAndName*>(doris::vectorized::ColumnWithTypeAndName*, doris::vectorized::ColumnWithTypeAndName*) /var/local/ldb_toolchain/include/c++/11/bits/stl_construct.h:152
#8 0x558c91635cd3 in void std::_Destroy<doris::vectorized::ColumnWithTypeAndName*>(doris::vectorized::ColumnWithTypeAndName*, doris::vectorized::ColumnWithTypeAndName*) /var/local/ldb_toolchain/include/c++/11/bits/stl_construct.h:185
#9 0x558c9163456c in void std::_Destroy<doris::vectorized::ColumnWithTypeAndName*, doris::vectorized::ColumnWithTypeAndName>(doris::vectorized::ColumnWithTypeAndName*, doris::vectorized::ColumnWithTypeAndName*, std::allocator<doris::vectorized::ColumnWithTypeAndName>&) /var/local/ldb_toolchain/include/c++/11/bits/alloc_traits.h:746
#10 0x558c9163292b in std::vector<doris::vectorized::ColumnWithTypeAndName, std::allocator<doris::vectorized::ColumnWithTypeAndName> >::~vector() /var/local/ldb_toolchain/include/c++/11/bits/stl_vector.h:680
#11 0x558c91630d63 in doris::vectorized::Block::~Block() /home/zcp/repo_center/doris_master/be/src/vec/core/block.h:60
#12 0x558c91ac66d2 in void doris::MemTable::_collect_vskiplist_results<false>() /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:345
#13 0x558c91ab3209 in doris::MemTable::shrink_memtable_by_agg() /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:351
#14 0x558c92c17605 in doris::DeltaWriter::write(doris::vectorized::Block const*, std::vector<int, std::allocator<int> > const&) /home/zcp/repo_center/doris_master/be/src/olap/delta_writer.cpp:202
#15 0x558c92e183b7 in doris::Status doris::TabletsChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) /home/zcp/repo_center/doris_master/be/src/runtime/tablets_channel.h:218
#16 0x558c92e14859 in doris::Status doris::LoadChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) /home/zcp/repo_center/doris_master/be/src/runtime/load_channel.h:152
#17 0x558c92e0dfe7 in doris::Status doris::LoadChannelMgr::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) /home/zcp/repo_center/doris_master/be/src/runtime/load_channel_mgr.h:134
#18 0x558c92df037e in operator() /home/zcp/repo_center/doris_master/be/src/service/internal_service.cpp:256
#19 0x558c92dfcadf in __invoke_impl<void, doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, const doris::PTabletWriterAddBlockRequest*, doris::PTabletWriterAddBl

ockResult, google::protobuf::Closure)::<lambda()>&> /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61

20 0x558c92dfc5c7 in __invoke_r<void, doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController, const doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult, google::protobuf::Closure)::<lambda()>&> /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111

#21 0x558c92dfc04b in _M_invoke /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#22 0x558c92876551 in std::function<void ()>::operator()() const /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#23 0x558c9287215c in doris::PriorityThreadPool::work_thread(int) /home/zcp/repo_center/doris_master/be/src/util/priority_thread_pool.hpp:136
#24 0x558c92888edf in void std::__invoke_impl<void, void (doris::PriorityThreadPool::* const&)(int), doris::PriorityThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityThreadPool::*`
dataroaring commented 2 years ago

`previously allocated by thread T707 here:

0 0x558c90fd0707 in operator new(unsigned long) (/mnt/ssd01/doris-master/VEC_ASAN/be/lib/doris_be+0x9189707)

#1 0x558c913e56b5 in COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnNullable> COWHelper<doris::vectorized::IColumn, doris::vectorized::ColumnNullable>::create<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnVector<unsigned char> > >(COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnVector<unsigned char> >&&) /home/zcp/repo_center/doris_master/be/src/vec/common/cow.h:412
#2 0x558c913db57c in COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnNullable> doris::vectorized::ColumnNullable::create<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnVector<unsigned char> >, void>(COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&&, COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::ColumnVector<unsigned char> >&&) /home/zcp/repo_center/doris_master/be/src/vec/columns/column_nullable.h:67
#3 0x558c9431fe74 in doris::vectorized::DataTypeNullable::create_column() const /home/zcp/repo_center/doris_master/be/src/vec/data_types/data_type_nullable.cpp:132
#4 0x558c942b03bf in doris::vectorized::Block::mutate_columns() /home/zcp/repo_center/doris_master/be/src/vec/core/block.cpp:461
#5 0x558c91abf926 in doris::vectorized::MutableBlock::MutableBlock(doris::vectorized::Block*) /home/zcp/repo_center/doris_master/be/src/vec/core/block.h:346
#6 0x558c91abf889 in doris::vectorized::MutableBlock::build_mutable_block(doris::vectorized::Block*) /home/zcp/repo_center/doris_master/be/src/vec/core/block.h:338
#7 0x558c91ab0e75 in doris::MemTable::insert(doris::vectorized::Block const*, std::vector<int, std::allocator<int> > const&) /home/zcp/repo_center/doris_master/be/src/olap/memtable.cpp:163
#8 0x558c92c175c3 in doris::DeltaWriter::write(doris::vectorized::Block const*, std::vector<int, std::allocator<int> > const&) /home/zcp/repo_center/doris_master/be/src/olap/delta_writer.cpp:199
#9 0x558c92e183b7 in doris::Status doris::TabletsChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) /home/zcp/repo_center/doris_master/be/src/runtime/tablets_channel.h:218
#10 0x558c92e14859 in doris::Status doris::LoadChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) /home/zcp/repo_center/doris_master/be/src/runtime/load_channel.h:152
#11 0x558c92e0dfe7 in doris::Status doris::LoadChannelMgr::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult*) /home/zcp/repo_center/doris_master/be/src/runtime/load_channel_mgr.h:134
#12 0x558c92df037e in operator() /home/zcp/repo_center/doris_master/be/src/service/internal_service.cpp:256
#13 0x558c92dfcadf in __invoke_impl<void, doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, const doris::PTabletWriterAddBlockRequest*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::<lambda()>&> /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:61
#14 0x558c92dfc5c7 in __invoke_r<void, doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController*, const doris::PTabletWriterAddBlockRequest*, doris::PTabletWriterAddBlockResult*, google::protobuf::Closure*)::<lambda()>&> /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:111
#15 0x558c92dfc04b in _M_invoke /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:291
#16 0x558c92876551 in std::function<void ()>::operator()() const /var/local/ldb_toolchain/include/c++/11/bits/std_function.h:560
#17 0x558c9287215c in doris::PriorityThreadPool::work_thread(int) /home/zcp/repo_center/doris_master/be/src/util/priority_thread_pool.hpp:136
#18 0x558c92888edf in void std::__invoke_impl<void, void (doris::PriorityThreadPool::* const&)(int), doris::PriorityThreadPool*&, int&>(std::__invoke_memfun_deref, void (doris::PriorityThreadPool::* const&)(int), doris::PriorityThreadPool*&, int&) /var/local/ldb_toolchain/include/c++/11/bits/invoke.h:74`