search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.77k stars 3.29k forks source link

[Bug] 2.0.2倒排搜索时服务异常退出 #26864

Open cst4049 opened 1 year ago

cst4049 commented 1 year ago

Search before asking

Version

2.0.2

What's Wrong?

doris 搜索数据 使用了match_all 时搜索be runtime error 直接重启

W1113 03:30:05.609741 602 scanner_scheduler.cpp:346] Scan thread read VScanner failed: [CANCELLED]cancelled W1113 03:30:05.609890 569 scanner_scheduler.cpp:346] Scan thread read VScanner failed: [CANCELLED]cancelled W1113 03:30:05.610100 360 fragment_mgr.cpp:481] report error status: [CANCELLED] to coordinator: TNetworkAddress(hostname=doris-fe-cluster-0.doris-fe-cluster.default.svc.cluster.local, port=9020), query id: b50c560ee9414546-8b228ffa61e28634, instance id: b50c560ee9414546-8b228ffa61e2863e W1113 03:30:05.610113 356 fragment_mgr.cpp:481] report error status: [CANCELLED] to coordinator: TNetworkAddress(hostname=doris-fe-cluster-0.doris-fe-cluster.default.svc.cluster.local, port=9020), query id: 28792c592ae24515-a17547eaee5b7a49, instance id: 28792c592ae24515-a17547eaee5b7a57 W1113 03:30:05.612196 356 status.h:383] meet error status: [RUNTIME_ERROR]

  1. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000ba1f197 in /opt/apache-doris/be/lib/doris_be
  2. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000ba1d72d in /opt/apache-doris/be/lib/doris_be
  3. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::Status doris::Status::Error(int, std::basic_string_view<char, std::char_traits >) @ 0x000000000aeb6e2b in /opt/apache-doris/be/lib/doris_be
  4. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::Status::create(doris::TStatus const&) @ 0x000000000b6f6e3e in /opt/apache-doris/be/lib/doris_be
  5. /root/src/doris-2.0/be/src/common/status.h:348: doris::FragmentMgr::coordinator_callback(doris::ReportStatusRequest const&) @ 0x000000000b81919e in /opt/apache-doris/be/lib/doris_be
  6. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:244: doris::pipeline::PipelineFragmentContext::send_report(bool) @ 0x000000001265d71f in /opt/apache-doris/be/lib/doris_be
  7. /root/src/doris-2.0/be/src/pipeline/pipeline_fragment_context.cpp:361: doris::pipeline::PipelineFragmentContext::_close_action() @ 0x000000001265e203 in /opt/apache-doris/be/lib/doris_be
  8. __pthread_once_slow @ 0x000000000001134f in /lib/x86_64-linux-gnu/libpthread-2.31.so
  9. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/mutex:783: doris::pipeline::PipelineFragmentContext::close_a_pipeline() @ 0x000000001265e119 in /opt/apache-doris/be/lib/doris_be
  10. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::pipeline::TaskScheduler::_try_close_task(doris::pipeline::PipelineTask*, doris::pipeline::PipelineTaskState) @ 0x00000000126ab9ef in /opt/apache-doris/be/lib/doris_be
  11. /root/src/doris-2.0/be/src/pipeline/task_scheduler.cpp:0: doris::pipeline::TaskScheduler::_do_work(unsigned long) @ 0x00000000126ab030 in /opt/apache-doris/be/lib/doris_be
  12. /root/src/doris-2.0/be/src/util/threadpool.cpp:0: doris::ThreadPool::dispatch_thread() @ 0x000000000ba5bdaf in /opt/apache-doris/be/lib/doris_be
  13. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba51d3c in /opt/apache-doris/be/lib/doris_be
  14. start_thread @ 0x0000000000008ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so
  15. clone @ 0x00000000000fddef in /lib/x86_64-linux-gnu/libc-2.31.so W1113 03:30:05.612255 360 status.h:383] meet error status: [RUNTIME_ERROR]
  16. /root/src/doris-2.0/be/src/common/stack_trace.cpp:302: StackTrace::tryCapture() @ 0x000000000ba1f197 in /opt/apache-doris/be/lib/doris_be
  17. /root/src/doris-2.0/be/src/common/stack_trace.h:0: doris::get_stack_trace[abi:cxx11]() @ 0x000000000ba1d72d in /opt/apache-doris/be/lib/doris_be
  18. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::Status doris::Status::Error(int, std::basic_string_view<char, std::char_traits >) @ 0x000000000aeb6e2b in /opt/apache-doris/be/lib/doris_be
  19. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187: doris::Status::create(doris::TStatus const&) @ 0x000000000b6f6e3e in /opt/apache-doris/be/lib/doris_be
  20. /root/src/doris-2.0/be/src/common/status.h:348: doris::FragmentMgr::coordinator_callback(doris::ReportStatusRequest const&) @ 0x000000000b81919e in /opt/apache-doris/be/lib/doris_be
  21. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:244: doris::pipeline::PipelineFragmentContext::send_report(bool) @ 0x000000001265d71f in /opt/apache-doris/be/lib/doris_be
  22. /root/src/doris-2.0/be/src/pipeline/pipeline_fragment_context.cpp:361: doris::pipeline::PipelineFragmentContext::_close_action() @ 0x000000001265e203 in /opt/apache-doris/be/lib/doris_be
  23. __pthread_once_slow @ 0x000000000001134f in /lib/x86_64-linux-gnu/libpthread-2.31.so
  24. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/mutex:783: doris::pipeline::PipelineFragmentContext::close_a_pipeline() @ 0x000000001265e119 in /opt/apache-doris/be/lib/doris_be
  25. /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:360: doris::pipeline::TaskScheduler::_try_close_task(doris::pipeline::PipelineTask*, doris::pipeline::PipelineTaskState) @ 0x00000000126ab9ef in /opt/apache-doris/be/lib/doris_be
  26. /root/src/doris-2.0/be/src/pipeline/task_scheduler.cpp:0: doris::pipeline::TaskScheduler::_do_work(unsigned long) @ 0x00000000126ab030 in /opt/apache-doris/be/lib/doris_be
  27. /root/src/doris-2.0/be/src/util/threadpool.cpp:0: doris::ThreadPool::dispatch_thread() @ 0x000000000ba5bdaf in /opt/apache-doris/be/lib/doris_be
  28. /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562: doris::Thread::supervise_thread(void*) @ 0x000000000ba51d3c in /opt/apache-doris/be/lib/doris_be
  29. start_thread @ 0x0000000000008ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so
  30. clone @ 0x00000000000fddef in /lib/x86_64-linux-gnu/libc-2.31.so I1113 03:30:11.539741 859 task_worker_pool.cpp:1068] successfully report TASK|host=doris-fe-cluster-0.doris-fe-cluster.default.svc.cluster.local|port=9020 I1113 03:30:15.109045 763 lru_cache_policy.h:69] DataPageCache prune stale 0 entries, 0 bytes, 99 times prune I1113 03:30:15.109316 763 lru_cache_policy.h:69] IndexPageCache prune stale 7 entries, 0 bytes, 90 times prune I1113 03:30:15.109400 763 lru_cache_policy.h:69] InvertedIndexSearcherCache prune stale 0 entries, 0 bytes, 111 times prune I1113 03:30:22.072539 1470 backend_service.cpp:363] get_batch stream_load_record rocksdb successfully. records size: 0, last_stream_load_timestamp: -1 I1113 03:30:24.548125 612 load_channel_mgr.cpp:250] cleaning timed out load channels I1113 03:30:24.548166 612 load_channel_mgr.cpp:282] load mem consumption(bytes). limit: 11747729817, current: 0, peak: 0, total running load channels: 0 I1113 03:30:25.046125 771 olap_server.cpp:1064] cooldown producer get tablet num: 0 I1113 03:30:25.109679 763 lru_cache_policy.h:69] DataPageCache prune stale 0 entries, 0 bytes, 100 times prune I1113 03:30:25.109768 763 lru_cache_policy.h:69] InvertedIndexSearcherCache prune stale 0 entries, 0 bytes, 112 times prune I1113 03:30:25.541183 859 task_worker_pool.cpp:1068] successfully report TASK|host=doris-fe-cluster-0.doris-fe-cluster.default.svc.cluster.local|port=9020 I1113 03:30:28.548206 1429 heartbeat_server.cpp:61] get heartbeat from FE.host:doris-fe-cluster-0.doris-fe-cluster.default.svc.cluster.local, port:9020, cluster id:1846465649, counter:217, BE start time: 1699845085109

What You Expected?

正常搜索到数据

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

LemonLiTree commented 1 year ago

Can you provide a reproduction case?