[Bug] string column is too large:total_length=4297083314,element_number=162668

wuweiqing-dev commented 1 year ago

Search before asking

[X] I had searched in the issues and found no similar issues.

Version

doris-1.2.4-rc01

What's Wrong?

The BE node is inactivated and the OUT log reports an error:string column is too large:total_length=4297083314,element_number=162668

What You Expected?

In response to this situation, it is possible to make alarms, rather than direct inactivation of the BE node, which will cause serious consequences and the entire cluster will not work for a short time

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

wuweiqing-dev commented 1 year ago

F0519 22:01:45.518649 71394 column_string.h:66] string column length is too large: total_length=4297083314 ,element_number=162668 Check failure stack trace: @ 0x55b7b57ca2ad google::LogMessage::Fail() @ 0x55b7b57cc7e9 google::LogMessage::SendToLog() @ 0x55b7b57c9e16 google::LogMessage::Flush() @ 0x55b7b57cce59 google::LogMessageFatal::~LogMessageFatal() @ 0x55b7b13db145 doris::vectorized::ColumnString::insert_from() @ 0x55b7b13d188b doris::vectorized::ColumnString::insert_indices_from() @ 0x55b7b13c5408 doris::vectorized::ColumnNullable::insert_indices_from() @ 0x55b7b146badd doris::vectorized::MutableBlock::add_rows() @ 0x55b7afe7860c doris::MemTable::insert() @ 0x55b7aff5cd6e doris::DeltaWriter::write() @ 0x55b7b0450fff doris::TabletsChannel::add_batch<>() @ 0x55b7b0525963 doris::LoadChannel::add_batch<>() @ 0x55b7b0525b80 doris::LoadChannelMgr::add_batch<>() @ 0x55b7b051745a _ZNSt17_Function_handlerIFvvEZN5doris20PInternalServiceImpl24_tablet_writer_add_blockEPN6google8protobuf13RpcControllerEPKNS1_28PTabletWriterAddBlockRequestEPNS1_27PTabletWriterAddBlockResultEPNS4_7ClosureEEUlvE_E9_M_invokeERKSt9_Any_data @ 0x55b7b03309e5 doris::PriorityThreadPool::work_thread() @ 0x55b7b8208b10 execute_native_thread_routine @ 0x7f0059a4adc5 start_thread @ 0x7f0059d5673d clone @ (nil) (unknown) Query id: 0-0 Aborted at 1684504906 (unix time) try "date -d @1684504906" if you are using GNU date Current BE git commitID: Unknown SIGABRT unkown detail explain (@0x3e800010e4b) received by PID 69195 (TID 0x7efbb3f54700) from PID 69195; stack trace: 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t, void) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F0059C94250 in /lib64/libc.so.6 2# raise in /lib64/libc.so.6 3# GI_abort in /lib64/libc.so.6 4# 0x000055B7B57D4C99 in /opt/appdata/disk01/doris/be/lib/doris_be 5# 0x000055B7B57CA2AD in /opt/appdata/disk01/doris/be/lib/doris_be 6# google::LogMessage::SendToLog() in /opt/appdata/disk01/doris/be/lib/doris_be 7# google::LogMessage::Flush() in /opt/appdata/disk01/doris/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /opt/appdata/disk01/doris/be/lib/doris_be 9# doris::vectorized::ColumnString::insert_from(doris::vectorized::IColumn const&, unsigned long) at /root/doris/be/src/vec/columns/column_string.h:151 10# doris::vectorized::ColumnString::insert_indices_from(doris::vectorized::IColumn const&, int const, int const) at /root/doris/be/src/vec/columns/column_string.cpp:106 11# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, int const, int const) at /root/doris/be/src/vec/columns/column_nullable.cpp:259 12# doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const, int const, int const) at /root/doris/be/src/vec/core/block.cpp:1008 13# doris::MemTable::insert(doris::vectorized::Block const, std::vector<int, std::allocator > const&) at /root/doris/be/src/olap/memtable.cpp:202 14# doris::DeltaWriter::write(doris::vectorized::Block const, std::vector<int, std::allocator > const&) at /root/doris/be/src/olap/delta_writer.cpp:233 15# doris::Status doris::TabletsChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequestconst&, doris::PTabletWriterAddBlockResult) at /root/doris/be/src/runtime/tablets_channel.cpp:489 16# doris::Status doris::LoadChannel::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequest const&, doris::PTabletWriterAddBlockResult) at /root/doris/be/src/runtime/load_channel.h:163 17# doris::Status doris::LoadChannelMgr::add_batch<doris::PTabletWriterAddBlockRequest, doris::PTabletWriterAddBlockResult>(doris::PTabletWriterAddBlockRequestconst&, doris::PTabletWriterAddBlockResult) at /root/doris/be/src/runtime/load_channel_mgr.h:168 18# std::_Function_handler<void (), doris::PInternalServiceImpl::_tablet_writer_add_block(google::protobuf::RpcController, doris::PTabletWriterAddBlockRequestconst, doris::PTabletWriterAddBlockResult, google::protobuf::Closure)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291 19# doris::PriorityThreadPool::work_thread(int) at /root/doris/be/src/util/priority_thread_pool.hpp:140 20# execute_native_thread_routine in /opt/appdata/disk01/doris/be/lib/doris_be 21# start_thread in /lib64/libpthread.so.0 22# clone in /lib64/libc.so.6

dataroaring commented 1 year ago

How to reproduce it?

JJJzheng commented 1 year ago

遇到了同样的问题，随机出现的。出现了 crash 之后，be 节点就挂了。

michael1991 commented 1 year ago

Facing same issue here ...

davycun commented 1 year ago

2、查询这个表的json 字段的时候就报错 select extra from my_catalog.my_schema.my_table limit 1;

3、下面执行sql后，backend的异常信息（string column length is too large） doris-be01-1 | terminate called after throwing an instance of 'doris::Exception' doris-be01-1 | what(): [E-3113] string column length is too large: total_length=94751273517062, element_number=1 doris-be01-1 | 0. /root/src/doris/be/src/common/stack_trace.cpp:298: StackTrace::tryCapture() @ 0x000000000b36ece7 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 1. /root/src/doris/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000b36d2bd in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 2. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173: doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits >) @ 0x000000000ae1c7ae in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::Exception::Exception<unsigned long&, unsigned long&>(int, std::basic_string_view<char, std::char_traits >, unsigned long&, unsigned long&) @ 0x000000000a8e2818 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 4. /root/src/doris/be/src/vec/columns/column_string.h:71: doris::vectorized::ColumnString::insert(doris::vectorized::Field const&) @ 0x000000000c3c725b in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 5. /root/src/doris/be/src/vec/common/cow.h:402: doris::vectorized::IDataType::create_column_const(unsigned long, doris::vectorized::Field const&) const @ 0x000000000d1004a2 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 6. /root/src/doris/be/src/vec/data_types/data_type.cpp:0: doris::vectorized::IDataType::create_column_const_with_default_value(unsigned long) const @ 0x000000000d100596 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 7. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:98: doris::vectorized::JdbcConnector::_cast_string_to_json(doris::SlotDescriptor const, doris::vectorized::Block, int, int) @ 0x000000000e7c2028 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 8. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::vectorized::JdbcConnector::get_next(bool, std::vector<COW::mutable_ptr, std::allocator<COW::mutable_ptr > >&, doris::vectorized::Block, int) @ 0x000000000e7bf7ef in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 9. /root/src/doris/be/src/common/status.h:414: doris::vectorized::NewJdbcScanner::_get_block_impl(doris::RuntimeState, doris::vectorized::Block, bool) @ 0x000000000e5f3caa in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 10. /root/src/doris/be/src/vec/exec/scan/vscanner.cpp:0: doris::vectorized::VScanner::get_block(doris::RuntimeState, doris::vectorized::Block, bool) @ 0x000000000e6f8584 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 11. /root/src/doris/be/src/common/status.h:335: doris::vectorized::ScannerScheduler::_scanner_scan(doris::vectorized::ScannerScheduler, doris::vectorized::ScannerContext, std::shared_ptr) @ 0x000000000e658723 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 12. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:701: std::_Function_handler<void (), doris::vectorized::ScannerScheduler::_schedule_scanners(doris::vectorized::ScannerContext)::$_1::operator()() const::{lambda()#3}>::_M_invoke(std::_Any_data const&) @ 0x000000000e6599a1 in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 13. /root/src/doris/be/src/util/threadpool.cpp:0: doris::ThreadPool::dispatch_thread() @ 0x000000000b3ab4bf in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 14. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void) @ 0x000000000b3a153c in /opt/apache-doris/be/lib/doris_be doris-be01-1 | 15. start_thread @ 0x0000000000008ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so doris-be01-1 | 16. clone @ 0x00000000000fddef in /lib/x86_64-linux-gnu/libc-2.31.so

haoshuai5358 commented 1 year ago

我在1.2.6也遇到了同样的问题 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t, void) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007F0059C94250 in /lib64/libc.so.6 2# raise in /lib64/libc.so.6 3# __GI_abort in /lib64/libc.so.6 4# 0x000055B7B57D4C99 in /opt/appdata/disk01/doris/be/lib/doris_be 5# 0x000055B7B57CA2AD in /opt/appdata/disk01/doris/be/lib/doris_be 6# google::LogMessage::SendToLog() in /opt/appdata/disk01/doris/be/lib/doris_be 7# google::LogMessage::Flush() in /opt/appdata/disk01/doris/be/lib/doris_be 8# google::LogMessageFatal::~LogMessageFatal() in /opt/appdata/disk01/doris/be/lib/doris_be 9# doris::vectorized::ColumnString::insert_from(doris::vectorized::IColumn const&, unsigned long) at /root/doris/be/src/vec/columns/column_string.h:151 10# doris::vectorized::ColumnString::insert_indices_from(doris::vectorized::IColumn const&, int const, int const) at /root/doris/be/src/vec/columns/column_string.cpp:106 11# doris::vectorized::ColumnNullable::insert_indices_from(doris::vectorized::IColumn const&, int const, int const) at /root/doris/be/src/vec/columns/column_nullable.cpp:259 12# doris::vectorized::MutableBlock::add_rows(doris::vectorized::Block const, int const, int const) at /root/doris/be/src/vec/core/block.cpp:1008 13# doris::MemTable::insert(doris::vectorized::Block const, std::vector<int, std::allocator > const&) at /root/doris/be/src/olap/memtable.cpp:202