search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.3k stars 3.21k forks source link

[Bug] 1.2.6， after search 4700000 log，be broken，and can not start anymore #23032

Open ouyangkui opened 1 year ago

ouyangkui commented 1 year ago

Search before asking

Version

1.2.6 centos7 8c 24g

What's Wrong?

after search 4700000 log，be broken，and can not start anymore

I0816 09:48:31.831986 78456 task_worker_pool.cpp:1523] successfully report TABLET|host=127.0.0.1|port=9020 terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc Query id: 0-0 Aborted at 1692150511 (unix time) try "date -d @1692150511" if you are using GNU date Current BE git commitID: Unknown SIGABRT unkown detail explain (@0x12f52) received by PID 77650 (TID 0x7fd7d85d7700) from PID 77650; stack trace: I0816 09:48:32.293300 77689 daemon.cpp:233] OS physical memory 23.37 GB. Process memory usage 409.59 MB, limit 18.69 GB, soft limit 16.83 GB. Sys available memory 15.18 GB, low water mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B I0816 09:48:33.202134 77689 daemon.cpp:233] OS physical memory 23.37 GB. Process memory usage 520.55 MB, limit 18.69 GB, soft limit 16.83 GB. Sys available memory 15.09 GB, low water mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t, void) at /root/doris/be/src/common/signal_handler.h:420 1# 0x00007FD95F0B4400 in /lib64/libc.so.6 2# gsignal in /lib64/libc.so.6 3# abort in /lib64/libc.so.6 4# __gnu_cxx::verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75 5# cxxabiv1::__terminate(void ()()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48 6# 0x0000558A70F3D581 in /data/apache-doris-1.2.6-bin-x64-noavx2/be/lib/doris_be 7# 0x0000558A70F3D6D4 in /data/apache-doris-1.2.6-bin-x64-noavx2/be/lib/doris_be 8# 0x0000558A70F3DAC6 in /data/apache-doris-1.2.6-bin-x64-noavx2/be/lib/doris_be 9# doris::faststring::GrowArray(unsigned long) at /root/doris/be/src/util/faststring.cc:41 10# doris::segment_v2::BinaryDictPageBuilder::add(unsigned char const, unsigned long) at /root/doris/be/src/olap/rowset/segment_v2/binary_dict_page.cpp:68 11# doris::segment_v2::ScalarColumnWriter::append_data_in_current_page(unsigned char const, unsigned long*) at /root/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:340 12# doris::segment_v2::ScalarColumnWriter::append_data(unsigned char const*, unsigned long) at /root/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:328 13# doris::segment_v2::ColumnWriter::append_nullable(unsigned char const, unsigned char const*, unsigned long) at /root/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:219 14# doris::segment_v2::ColumnWriter::append(unsigned char const, void const, unsigned long) at /root/doris/be/src/olap/rowset/segment_v2/column_writer.cpp:235 15# doris::segment_v2::SegmentWriter::append_block(doris::vectorized::Block const, unsigned long, unsigned long) at /root/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:200 16# doris::BetaRowsetWriter::_add_block(doris::vectorized::Block const, std::unique_ptr<doris::segment_v2::SegmentWriter, std::default_delete >) at /root/doris/be/src/olap/rowset/beta_rowset_writer.cpp:576 17# doris::BetaRowsetWriter::add_block(doris::vectorized::Block const) at /root/doris/be/src/olap/rowset/beta_rowset_writer.cpp:121 18# doris::VSchemaChangeDirectly::_inner_process(std::shared_ptr, doris::RowsetWriter, std::shared_ptr, std::shared_ptr) at /root/doris/be/src/olap/schema_change.cpp:1345 19# doris::SchemaChange::process(std::shared_ptr, doris::RowsetWriter, std::shared_ptr, std::shared_ptr, std::shared_ptr) at /var/local/ldb-toolchain/include/c++/11/bits/shared_ptr.h:122 20# doris::SchemaChangeHandler::_convert_historical_rowsets(doris::SchemaChangeHandler::SchemaChangeParams const&) at /root/doris/be/src/olap/schema_change.cpp:2229 21# doris::SchemaChangeHandler::_do_process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) at /root/doris/be/src/olap/schema_change.cpp:2022 22# doris::SchemaChangeHandler::process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) at /root/doris/be/src/olap/schema_change.cpp:1759 23# doris::EngineAlterTabletTask::execute() at /root/doris/be/src/olap/task/engine_alter_tablet_task.cpp:41 24# doris::StorageEngine::execute_task(doris::EngineTask) at /root/doris/be/src/olap/storage_engine.cpp:1013 25# doris::TaskWorkerPool::_alter_tablet(doris::TAgentTaskRequest const&, long, doris::TTaskType::type, doris::TFinishTaskRequest) at /root/doris/be/src/agent/task_worker_pool.cpp:536 26# doris::TaskWorkerPool::_alter_tablet_worker_thread_callback() at /root/doris/be/src/agent/task_worker_pool.cpp:496 27# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:543 28# doris::Thread::supervise_thread(void) at /root/doris/be/src/util/thread.cpp:455 29# start_thread in /lib64/libpthread.so.0 30# clone in /lib64/libc.so.6

./start_be.sh: 行 321: 77650 已放弃 ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" 2>&1 < /dev/null

What You Expected?

fixed

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

ouyangkui commented 1 year ago

reproduce this issue with 2.0. upload csv log, by sql, insert 1000 logs every times,

create table invertedIneex ( dt DATETIMEV2(3), clientip VARCHAR(20), user TEXT, method VARCHAR(20), request TEXT, protocol VARCHAR(20), status int(11), bytes int(10), referer TEXT, user_agent TEXT, x_forwarded_for TEXT, raw_message TEXT, INDEX index_raw (raw_message) USING INVERTED PROPERTIES("parser" = "english") COMMENT 'reindex' )

DUPLICATE KEY(dt) DISTRIBUTED BY HASH(dt) BUCKETS 1 PROPERTIES ( "replication_num" = "1" )

1 log example ::

2023-01-15 01:52:58.165000,113.128.11.195,-,GET,/index/login/?gw_address=192.168.11.1&gw_port=2060&gw_id=0531HY000123&mac=d4:97:0b:15:e3:5e&url=http%3A//infoc2.duba.net/c/,HTTP/1.1,200,4633,-,-,-,113.128.11.195 - - [15/Jan/2023:01:52:58.165 +0800] "GET /index/login/?gw_address=192.168.11.1&gw_port=2060&gw_id=0531HY000123&mac=d4:97:0b:15:e3:5e&url=http%3A//infoc2.duba.net/c/ HTTP/1.1" 200 4633 "-" "-" "-"

result: be crashed, and can not start anymore

I0821 15:17:02.644104 6323 tablets_channel.cpp:103] open tablets channel: (load_id=26176df166004912-a4ebb62a9444d314, index_id=18007), tablets num: 1, timeout(s): 14400 I0821 15:17:02.668521 5702 vtablet_sink.cpp:894] VNodeChannel[18007-10034], load_id=26176df166004912-a4ebb62a9444d314, txn_id=7004, node=192.168.40.117:8060 mark closed, left pending batch size: 1 I0821 15:17:02.671339 6409 vtablet_sink.cpp:1121] all node channels are stopped(maybe finished/offending/cancelled), sender thread exit. 26176df166004912-a4ebb62a9444d314 I0821 15:17:02.674144 6185 tablets_channel.cpp:145] close tablets channel: (load_id=26176df166004912-a4ebb62a9444d314, index_id=18007), sender id: 0, backend id: 10034 Query id: 0-0 Aborted at 1692602222 (unix time) try "date -d @1692602222" if you are using GNU date Current BE git commitID: 96bb9a7 SIGILL illegal operand (@0x555d7c8a21d0) received by PID 5387 (TID 5978 OR 0x7f9e37518700) from PID 2089427408; stack trace: I0821 15:17:03.537358 5424 daemon.cpp:218] OS physical memory 23.37 GB. Process memory usage 1.49 GB, limit 18.69 GB, soft limit 16.83 GB. Sys available memory 20.41 GB, low water mark 1.60 GB, warning water mark 3.20 GB. Refresh interval memory growth 0 B 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t, void) at /root/src/doris/be/src/common/signal_handler.h:413 1# os::Linux::chained_handler(int, siginfo_t, void) in /data/java/java-4.1.1.0/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /data/java/java-4.1.1.0/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo_t, void) in /data/java/java-4.1.1.0/jre/lib/amd64/server/libjvm.so 4# 0x00007F9F962E3400 in /lib64/libc.so.6 5# bitdienc32 at /root/src/doris/be/src/clucene/src/ext/for/bitutil.c:384 6# p4nd1enc128v32 at /root/src/doris/be/src/clucene/src/ext/for/vp4c.c:414 7# lucene::index::SDocumentsWriter::appendPostings(lucene::util::ArrayBase<lucene::index::SDocumentsWriter::ThreadState::FieldData>, lucene::index::STermInfosWriter, lucene::store::IndexOutput, lucene::store::IndexOutput) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 8# lucene::index::SDocumentsWriter::writeSegment(std::vector<std::cxx11::basic_string<char, std::char_traits, std::allocator >, std::allocator<std::cxx11::basic_string<char, std::char_traits, std::allocator > > >&) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 9# lucene::index::SDocumentsWriter::flush(bool) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 10# lucene::index::IndexWriter::doFlush(bool) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 11# lucene::index::IndexWriter::flush(bool, bool) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 12# lucene::index::IndexWriter::closeInternal(bool) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 13# lucene::index::IndexWriter::close(bool) in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 14# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)26>::close() at /root/src/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:110 15# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)26>::finish() at /root/src/doris/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:425 16# doris::segment_v2::ScalarColumnWriter::write_inverted_index() in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 17# doris::segment_v2::SegmentWriter::finalize_columns_index(unsigned long) at /root/src/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:872 18# doris::segment_v2::SegmentWriter::finalize(unsigned long, unsigned long) at /root/src/doris/be/src/olap/rowset/segment_v2/segment_writer.cpp:914 19# doris::BetaRowsetWriter::_flush_segment_writer(std::unique_ptr<doris::segment_v2::SegmentWriter, std::default_delete >, long) at /root/src/doris/be/src/olap/rowset/beta_rowset_writer.cpp:804 20# doris::BetaRowsetWriter::flush_single_memtable(doris::vectorized::Block const, long, doris::FlushContext const) at /root/src/doris/be/src/olap/rowset/beta_rowset_writer.cpp:506 21# doris::MemTable::_do_flush() at /root/src/doris/be/src/olap/memtable.cpp:540 22# doris::MemTable::flush() at /root/src/doris/be/src/olap/memtable.cpp:503 23# doris::FlushToken::_flush_memtable(doris::MemTable, long) at /root/src/doris/be/src/olap/memtable_flush_executor.cpp:99 24# doris::MemtableFlushTask::run() at /root/src/doris/be/src/olap/memtable_flush_executor.cpp:47 25# doris::ThreadPool::dispatch_thread() in /data/apache-doris-2.0.0-bin-x64-noavx2/be/lib/doris_be 26# doris::Thread::supervise_thread(void*) at /root/src/doris/be/src/util/thread.cpp:466 27# start_thread in /lib64/libpthread.so.0 28# clone in /lib64/libc.so.6

./start_be.sh: 行 325: 5387 非法指令 ${LIMIT:+${LIMIT}} "${DORIS_HOME}/lib/doris_be" "$@" 2>&1 < /dev/null