[Help] BE crash

[zh0122]

BE crash.

Pls help to check the log. Below is the logs(be.out) in different BE.

start time: Thu Jan 28 17:10:52 CST 2021
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
*** Aborted at 1611937804 (unix time) try "date -d @1611937804" if you are using GNU date ***
PC: @     0x7f1f69bbdf94 __memset_sse2
*** SIGSEGV (@0x0) received by PID 26638 (TID 0x7f1f445b7700) from PID 0; stack trace: ***
    @     0x7f1f69b65280 (unknown)
    @     0x7f1f69bbdf94 __memset_sse2
    @          0x154ffa7 doris::OlapScanner::get_batch()
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
    @          0x151aa95 doris::OlapScanNode::scanner_thread()
    @          0x100094e doris::PriorityThreadPool::work_thread()
src/central_freelist.cc:333] tcmalloc: allocation failed 131072
    @          0x1b3c2ed thread_proxy
    @     0x7f1f6991add5 start_thread
    @     0x7f1f69c2cead __clone
start time: Sat Jan 30 00:40:49 CST 2021
start time: Sat Jan 30 00:40:54 CST 2021
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
*** Aborted at 1612026606 (unix time) try "date -d @1612026606" if you are using GNU date ***
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
PC: @     0x7f3e35608207 __GI_raise
*** SIGABRT (@0x3ed00003497) received by PID 13463 (TID 0x7f3cc9e21700) from PID 13463; stack trace: ***
terminate called after throwing an instance of 'terminate called recursively
std::bad_allocterminate called recursively
terminate called recursively
'
terminate called recursively
  what():  std::bad_alloc
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
    @     0x7f3e35608280 (unknown)
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
    @     0x7f3e35608207 __GI_raise
    @     0x7f3e356098f8 __GI_abort
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
    @     0x7f3e35601026 __assert_fail_base
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
    @     0x7f3e356010d2 __GI___assert_fail
    @           0xbb3982 rd_malloc.part.0
    @          0x1a19d72 rd_kafka_lz4_decompress
    @          0x1a0a142 rd_kafka_msgset_reader_decompress
    @          0x1a0d4bd rd_kafka_msgset_reader_v2
    @          0x1a05e40 rd_kafka_msgset_reader_run
    @          0x1a0f00e rd_kafka_msgset_parse
    @          0x199e83c rd_kafka_fetch_reply_handle
    @          0x19a05c7 rd_kafka_broker_fetch_reply
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
terminate called recursively
    @          0x19bde5b rd_kafka_buf_callback
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
src/central_freelist.cc:333] tcmalloc: allocation failed 65536
terminate called recursively
    @          0x19a32f3 rd_kafka_recv
    @          0x19bba48 rd_kafka_transport_io_event
    @          0x19a9971 rd_kafka_broker_serve
    @          0x19aab8d rd_kafka_broker_thread_main
    @          0x19f9507 _thrd_wrapper_function
    @     0x7f3e353bddd5 start_thread
    @     0x7f3e356cfead __clone
start time: Sun Jan 31 01:21:16 CST 2021

start time: Thu Jan 28 17:11:16 CST 2021
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
terminate called recursively
terminate called after throwing an instance of 'terminate called recursively
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
*** Aborted at 1611940138 (unix time) try "date -d @1611940138" if you are using GNU date ***
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
std::bad_alloc'
  what():  std::bad_alloc
PC: @     0x7f30a8fd6207 __GI_raise
*** SIGABRT (@0x3ed00006e27) received by PID 28199 (TID 0x7f304b146700) from PID 28199; stack trace: ***
    @     0x7f30a8fd6280 (unknown)
    @     0x7f30a8fd6207 __GI_raise
    @     0x7f30a8fd78f8 __GI_abort
    @          0x275d09d __gnu_cxx::__verbose_terminate_handler()
    @          0x26c9196 __cxxabiv1::__terminate()
    @          0x26c91e1 std::terminate()
    @          0x26c7996 __cxa_rethrow
    @           0xf777a3 _ZNSt10_HashtableIN5doris5SliceESt4pairIKS1_jESaIS4_ENSt8__detail10_Select1stESt8equal_toIS1_ENS0_10segment_v221BinaryDictPageBuilder11HashOfSliceENS6_18_Mod_range_hashingENS6_20_Default_ranged_hashENS6_20_Prime_rehash_policyENS6_17_Hashtable_traitsILb1ELb0ELb1EEEE10_M_emplaceIJRS1_RjEEES2_INS6_14_Node_iteratorIS4_Lb0ELb1EEEbESt17integral_constantIbLb1EEDpOT_
    @           0xf76d94 doris::segment_v2::BinaryDictPageBuilder::add()
    @          0x16d0da1 doris::segment_v2::ScalarColumnWriter::append_data()
    @          0x16cdcd7 doris::segment_v2::ColumnWriter::append_nullable()
    @          0x16a292f doris::segment_v2::SegmentWriter::append_row<>()
    @           0xf9a0a6 doris::BetaRowsetWriter::_add_row<>()
    @           0xee1898 doris::Merger::merge_rowsets()
    @           0xed047a doris::Compaction::do_compaction_impl()
    @           0xed2c0c doris::Compaction::do_compaction()
    @           0xed3e91 doris::CumulativeCompaction::execute_compact_impl()
    @           0xeced24 doris::Compaction::execute_compact()
    @           0xe54fb6 doris::Tablet::execute_compaction()
    @           0xe2ef1f _ZNSt17_Function_handlerIFvvEZN5doris13StorageEngine35_compaction_tasks_producer_callbackEvEUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x11be4b2 doris::ThreadPool::dispatch_thread()
    @          0x11b6318 doris::Thread::supervise_thread()
    @     0x7f30a8d8bdd5 start_thread
    @     0x7f30a909dead __clone

start time: Sat Jan 30 01:16:24 CST 2021
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
*** Aborted at 1612026628 (unix time) try "date -d @1612026628" if you are using GNU date ***
PC: @     0x7ff9acad7207 __GI_raise
*** SIGABRT (@0x3ee0000029a) received by PID 666 (TID 0x7ff950d4a700) from PID 666; stack trace: ***
    @     0x7ff9acad7280 (unknown)
    @     0x7ff9acad7207 __GI_raise
    @     0x7ff9acad88f8 __GI_abort
    @          0x275d0d5 __gnu_cxx::__verbose_terminate_handler()
    @          0x26c9196 __cxxabiv1::__terminate()
    @          0x26c91e1 std::terminate()
    @          0x26c7943 __cxa_throw
    @          0x25f967c (anonymous namespace)::handle_oom()
    @          0x276404e tcmalloc::allocate_full_cpp_throw_oom()
    @          0x1192728 doris::faststring::GrowToAtLeast()
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
    @           0xf76aea doris::segment_v2::BinaryDictPageBuilder::get_dictionary_page()
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
    @          0x16cf325 doris::segment_v2::ScalarColumnWriter::write_data()
    @          0x16a23ef doris::segment_v2::SegmentWriter::finalize()
terminate called recursively
    @           0xf97fdf doris::BetaRowsetWriter::flush()
    @           0xee24bb doris::Merger::merge_rowsets()
    @           0xed047a doris::Compaction::do_compaction_impl()
    @           0xed2c0c doris::Compaction::do_compaction()
    @           0xed3e91 doris::CumulativeCompaction::execute_compact_impl()
    @           0xeced24 doris::Compaction::execute_compact()
    @           0xe54fb6 doris::Tablet::execute_compaction()
    @           0xe2ef1f _ZNSt17_Function_handlerIFvvEZN5doris13StorageEngine35_compaction_tasks_producer_callbackEvEUlvE0_E9_M_invokeERKSt9_Any_data
    @          0x11be4b2 doris::ThreadPool::dispatch_thread()
    @          0x11b6318 doris::Thread::supervise_thread()
    @     0x7ff9ac88cdd5 start_thread
    @     0x7ff9acb9eead __clone

start time: Sat Jan 30 01:16:26 CST 2021
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 8192
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 8192
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:95: rd_calloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:95: rd_calloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
*** Aborted at 1612026658 (unix time) try "date -d @1612026658" if you are using GNU date ***
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
terminate called recursively
terminate called after throwing an instance of 'palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
std::bad_alloc'
  what():  std::bad_alloc
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
PC: @     0x7ffa91d9f207 __GI_raise
*** SIGABRT (@0x3ed0002fc55) received by PID 195669 (TID 0x7ff9b674f700) from PID 195669; stack trace: ***
    @     0x7ffa91d9f280 (unknown)
    @     0x7ffa91d9f207 __GI_raise
    @     0x7ffa91da08f8 __GI_abort
    @     0x7ffa91d98026 __assert_fail_base
    @     0x7ffa91d980d2 __GI___assert_fail
    @           0xbb3982 rd_malloc.part.0
    @          0x1a19d72 rd_kafka_lz4_decompress
    @          0x1a0a142 rd_kafka_msgset_reader_decompress
    @          0x1a0d4bd rd_kafka_msgset_reader_v2
    @          0x1a05e40 rd_kafka_msgset_reader_run
    @          0x1a0f00e rd_kafka_msgset_parse
    @          0x199e83c rd_kafka_fetch_reply_handle
    @          0x19a05c7 rd_kafka_broker_fetch_reply
    @          0x19bde5b rd_kafka_buf_callback
    @          0x19a32f3 rd_kafka_recv
    @          0x19bba48 rd_kafka_transport_io_event
    @          0x19a9971 rd_kafka_broker_serve
    @          0x19aab8d rd_kafka_broker_thread_main
    @          0x19f9507 _thrd_wrapper_function
    @     0x7ffa91b54dd5 start_thread
    @     0x7ffa91e66ead __clone

start time: Thu Jan 28 17:04:56 CST 2021
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
*** Aborted at 1612026620 (unix time) try "date -d @1612026620" if you are using GNU date ***
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
terminate called recursively
terminate called recursively
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
src/central_freelist.cc:333] tcmalloc: allocation failed 24576
palo_be: rd.h:101: rd_malloc: Assertion `p' failed.
PC: @     0x7ff38e6d5207 __GI_raise
*** SIGABRT (@0x3eb00060c1e) received by PID 396318 (TID 0x7ff1d1163700) from PID 396318; stack trace: ***
    @     0x7ff38e6d5280 (unknown)
    @     0x7ff38e6d5207 __GI_raise
    @     0x7ff38e6d68f8 __GI_abort
    @     0x7ff38e6ce026 __assert_fail_base
    @     0x7ff38e6ce0d2 __GI___assert_fail
    @           0xbb3982 rd_malloc.part.0
    @          0x1a19d72 rd_kafka_lz4_decompress
    @          0x1a0a142 rd_kafka_msgset_reader_decompress
    @          0x1a0d4bd rd_kafka_msgset_reader_v2
    @          0x1a05e40 rd_kafka_msgset_reader_run
    @          0x1a0f00e rd_kafka_msgset_parse
    @          0x199e83c rd_kafka_fetch_reply_handle
    @          0x19a05c7 rd_kafka_broker_fetch_reply
    @          0x19bde5b rd_kafka_buf_callback
    @          0x19a32f3 rd_kafka_recv
    @          0x19bba48 rd_kafka_transport_io_event
    @          0x19a9971 rd_kafka_broker_serve
    @          0x19aab8d rd_kafka_broker_thread_main
    @          0x19f9507 _thrd_wrapper_function
    @     0x7ff38e48add5 start_thread
    @     0x7ff38e79cead __clone

[morningman]

Which version of Doris?

[zh0122]

Which version of Doris?

Hi,

I built the code from master branch in 2020.01.17. The git sha1 id is 73a67901edfe94b3083c376cade1122486fe4e28 [Metric] Add system memory metrics for fe (#5149), and cherry-pick with ea7f61e1c76502360da0b3f20a2f059fa13f2c56 [Bug] Duplicate results when reading aggregation table.

I don't know the reason to cause the crash. I saw some key words in log such as 'rd.h' , 'rd_kafka', but all the routine load are paused when the crash happened.

Pls help to check this.

Thx

[morningman]

I am not sure yet. Is there a case that can be reproduced stably?

[stdpain]

I advise to run a heap profile, to help analyze memory problems @zh0122 http://doris.incubator.apache.org/master/zh-CN/developer-guide/debug-tool.html#heap-profile