search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.39k stars 3.22k forks source link

节点掉线异常 #31024

Open allen-gf opened 7 months ago

allen-gf commented 7 months ago

Search before asking

Version

apache-doris-2.0.2

What's Wrong?

start time: Tue Feb 13 07:39:59 PM CST 2024 INFO: java_cmd /app/java/jdk/bin/java INFO: jdk_version 8 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/app/doris/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/preload-extensions/preload-extensions-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/app/doris/apache-doris-2.0.2-bin-x64/be/lib/java_extensions/java-udf/java-udf-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/app/doris/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Reload4jLoggerFactory] Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /app/doris/apache-doris-2.0.2-bin-x64/be/lib/hadoop_hdfs/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. Query id: 0-0 tablet id: 1440788 Aborted at 1707824410 (unix time) try "date -d @1707824410" if you are using GNU date Current BE git commitID: ae923f7 SIGSEGV unknown detail explain (@0x0) received by PID 2052478 (TID 2054316 OR 0x7f51072c4640) from PID 0; stack trace: 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t, void) at /root/src/doris-2.0/be/src/common/signal_handler.h:417 1# os::Linux::chained_handler(int, siginfo, void) in /app/java/jdk/jre/lib/amd64/server/libjvm.so 2# JVM_handle_linux_signal in /app/java/jdk/jre/lib/amd64/server/libjvm.so 3# signalHandler(int, siginfo, void) in /app/java/jdk/jre/lib/amd64/server/libjvm.so 4# 0x00007F5DC9971520 in /lib/x86_64-linux-gnu/libc.so.6 5# lucene::index::SDocumentsWriter::ThreadState::FieldData::addPosition(lucene::analysis::Token) in /app/doris/doris/be/lib/doris_be 6# lucene::index::SDocumentsWriter::ThreadState::FieldData::invertField(lucene::document::Field, lucene::analysis::Analyzer, int) in /app/doris/doris/be/lib/doris_be 7# lucene::index::SDocumentsWriter::ThreadState::FieldData::processField(lucene::analysis::Analyzer) in /app/doris/doris/be/lib/doris_be 8# lucene::index::SDocumentsWriter::ThreadState::processDocument(lucene::analysis::Analyzer) in /app/doris/doris/be/lib/doris_be 9# lucene::index::SDocumentsWriter::updateDocument(lucene::document::Document, lucene::analysis::Analyzer) in /app/doris/doris/be/lib/doris_be 10# lucene::index::IndexWriter::addDocument(lucene::document::Document, lucene::analysis::Analyzer) in /app/doris/doris/be/lib/doris_be 11# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::add_document() at /root/src/doris-2.0/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:224 12# doris::segment_v2::InvertedIndexColumnWriterImpl<(doris::FieldType)17>::add_values(std::__cxx11::basic_string<char, std::char_traits, std::allocator >, void const, unsigned long) at /root/src/doris-2.0/be/src/olap/rowset/segment_v2/inverted_index_writer.cpp:283 13# doris::segment_v2::ScalarColumnWriter::append_data_in_current_page(unsigned char const, unsigned long) at /root/src/doris-2.0/be/src/olap/rowset/segment_v2/column_writer.cpp:548 14# doris::segment_v2::ScalarColumnWriter::append_data(unsigned char const*, unsigned long) at /root/src/doris-2.0/be/src/olap/rowset/segment_v2/column_writer.cpp:528 15# doris::segment_v2::ColumnWriter::append_nullable(unsigned char const, unsigned char const*, unsigned long) at /root/src/doris-2.0/be/src/olap/rowset/segment_v2/column_writer.cpp:403 16# doris::segment_v2::ColumnWriter::append(unsigned char const, void const, unsigned long) in /app/doris/doris/be/lib/doris_be 17# doris::segment_v2::SegmentWriter::append_block(doris::vectorized::Block const, unsigned long, unsigned long) in /app/doris/doris/be/lib/doris_be 18# doris::VerticalBetaRowsetWriter::add_columns(doris::vectorized::Block const, std::vector<unsigned int, std::allocator > const&, bool, unsigned int) at /root/src/doris-2.0/be/src/olap/rowset/vertical_beta_rowset_writer.cpp:124 19# doris::Merger::vertical_compact_one_group(std::shared_ptr, doris::ReaderType, std::shared_ptr, bool, std::vector<unsigned int, std::allocator > const&, doris::vectorized::RowSourcesBuffer, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > const&, doris::RowsetWriter, long, doris::Merger::Statistics) at /root/src/doris-2.0/be/src/olap/merger.cpp:247 20# doris::Merger::vertical_merge_rowsets(std::shared_ptr, doris::ReaderType, std::shared_ptr, std::vector<std::shared_ptr, std::allocator<std::shared_ptr > > const&, doris::RowsetWriter, long, doris::Merger::Statistics) at /root/src/doris-2.0/be/src/olap/merger.cpp:351 21# doris::Compaction::do_compaction_impl(long) at /root/src/doris-2.0/be/src/olap/compaction.cpp:351 22# doris::Compaction::do_compaction(long) at /root/src/doris-2.0/be/src/olap/compaction.cpp:124 23# doris::CumulativeCompaction::execute_compact_impl() at /root/src/doris-2.0/be/src/olap/cumulative_compaction.cpp:87 24# doris::Compaction::execute_compact() at /root/src/doris-2.0/be/src/olap/compaction.cpp:106 25# doris::Tablet::execute_compaction(doris::CompactionType) at /root/src/doris-2.0/be/src/olap/tablet.cpp:1859 26# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptr, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291 27# doris::ThreadPool::dispatch_thread() in /app/doris/doris/be/lib/doris_be 28# doris::Thread::supervise_thread(void*) at /root/src/doris-2.0/be/src/util/thread.cpp:470 29# 0x00007F5DC99C3AC3 in /lib/x86_64-linux-gnu/libc.so.6 30# 0x00007F5DC9A55A40 in /lib/x86_64-linux-gnu/libc.so.6

What You Expected?

70+ 总是会掉一些节点，导致load任务失败，帮查原因

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

stalary commented 7 months ago

Are you use invert index?

allen-gf commented 7 months ago

你使用倒排索引吗？

yes

stalary commented 7 months ago

@xiaokang PTAL

xiaokang commented 7 months ago

@allen-gf can you provide you /proc/cpuinfo ?