tigerLVU
commented
1 year ago it works by 3 nodes, fe node works ok, 2 be nodes can't be work.
Open tigerLVU opened 1 year ago
tigerLVU
commented
1 year ago it works by 3 nodes, fe node works ok, 2 be nodes can't be work.
Search before asking
Version
1.1.1
What's Wrong?
be crash yesterday , can't start now .
I1027 12:54:03.003387 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/618/3061102/750553990 I1027 12:54:03.003896 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/619/3061107/887275578 I1027 12:54:03.004725 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/620/3061117/887275578 I1027 12:54:03.005306 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/621/3061113/887275578 I1027 12:54:03.005743 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/622/3061121/887275578 I1027 12:54:03.006450 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/623/3061125/887275578 I1027 12:54:03.007156 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/624/3061130/188199663 I1027 12:54:03.007730 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/625/3061136/188199663 I1027 12:54:03.008370 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/626/3061138/188199663 I1027 12:54:03.008949 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/627/3061142/188199663 I1027 12:54:03.009665 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/628/3061146/188199663 I1027 12:54:03.010108 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/629/3061157/479033467 I1027 12:54:03.010697 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/630/3061163/479033467 I1027 12:54:03.011034 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/631/3061172/1030160689 I1027 12:54:03.011380 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/632/3061174/1030160689 I1027 12:54:03.011706 29788 tablet_manager.cpp:1095] tablet meta exist is meta store, skip delete the path /data/storage/data/633/3061184/1030160689 I1027 12:54:03.255105 29788 data_dir.cpp:606] finished one time path gc by tablet. I1027 12:54:03.255184 29788 olap_server.cpp:293] try to perform path gc by rowsetid! I1027 12:54:03.255198 29788 data_dir.cpp:617] start to path gc by rowsetid. I1027 12:54:03.364904 29788 data_dir.cpp:716] collect garbage dir path: /data/storage/data/495/3065354/807381673/0200000000000001ed47684ce6b546d65227a984f3c4a5a0_0.dat I1027 12:54:03.365450 29788 data_dir.cpp:716] collect garbage dir path: /data/storage/data/497/3065358/807381673/0200000000000003ed47684ce6b546d65227a984f3c4a5a0_0.dat I1027 12:54:03.365653 29788 data_dir.cpp:716] collect garbage dir path: /data/storage/data/498/3065370/807381673/0200000000000002ed47684ce6b546d65227a984f3c4a5a0_0.dat I1027 12:54:03.473474 29788 data_dir.cpp:650] finished one time path gc by rowsetid. I1027 12:54:03.763403 29941 snapshot_manager.cpp:303] receive a make snapshot request, request detail is TSnapshotRequest { 01: tablet_id (i64) = 1522018, 02: schema_hash (i32) = 1452470451, 05: timeout (i64) = 180, 09: preferred_snapshot_version (i32) = 4, }, snapshot_version is 4 I1027 12:54:03.765173 29941 snapshot_manager.cpp:88] success to make snapshot. path=['/data/storage/snapshot/20231027125403.0.180'] I1027 12:54:03.765206 29941 agent_server.cpp:223] success to make_snapshot. tablet_id=1522018, schema_hash=1452470451, snapshot_path: /data/storage/snapshot/20231027125403.0.180 I1027 12:54:03.767205 29923 download_action.cpp:107] accept one download request HttpRequest: method:0 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125403.0.180//1522018/1452470451/ raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125403.0.180//1522018/1452470451/ key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:03.767485 29923 download_action.cpp:127] deal with download request finished! I1027 12:54:03.768604 29891 download_action.cpp:107] accept one download request HttpRequest: method:4 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125403.0.180//1522018/1452470451/1522018.hdr raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125403.0.180//1522018/1452470451/1522018.hdr key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:03.768774 29891 download_action.cpp:127] deal with download request finished! I1027 12:54:03.770607 29897 download_action.cpp:107] accept one download request HttpRequest: method:0 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125403.0.180//1522018/1452470451/1522018.hdr raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125403.0.180//1522018/1452470451/1522018.hdr key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:03.770843 29897 download_action.cpp:127] deal with download request finished! I1027 12:54:03.773475 29941 snapshot_manager.cpp:108] success to release snapshot path. [path='/data/storage/snapshot/20231027125403.0.180/'] I1027 12:54:03.773514 29941 agent_server.cpp:243] success to release_snapshot. snapshot_path=/data/storage/snapshot/20231027125403.0.180/, err_code=0 I1027 12:54:04.779556 29941 snapshot_manager.cpp:303] receive a make snapshot request, request detail is TSnapshotRequest { 01: tablet_id (i64) = 1200560, 02: schema_hash (i32) = 1210587744, 05: timeout (i64) = 180, 09: preferred_snapshot_version (i32) = 4, }, snapshot_version is 4 I1027 12:54:04.781397 29941 snapshot_manager.cpp:88] success to make snapshot. path=['/data/storage/snapshot/20231027125404.1.180'] I1027 12:54:04.781430 29941 agent_server.cpp:223] success to make_snapshot. tablet_id=1200560, schema_hash=1210587744, snapshot_path: /data/storage/snapshot/20231027125404.1.180 I1027 12:54:04.783692 29904 download_action.cpp:107] accept one download request HttpRequest: method:0 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125404.1.180//1200560/1210587744/ raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125404.1.180//1200560/1210587744/ key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:04.783934 29904 download_action.cpp:127] deal with download request finished! I1027 12:54:04.785353 29911 download_action.cpp:107] accept one download request HttpRequest: method:4 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125404.1.180//1200560/1210587744/1200560.hdr raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125404.1.180//1200560/1210587744/1200560.hdr key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:04.785557 29911 download_action.cpp:127] deal with download request finished! I1027 12:54:04.786824 29904 download_action.cpp:107] accept one download request HttpRequest: method:0 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125404.1.180//1200560/1210587744/1200560.hdr raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125404.1.180//1200560/1210587744/1200560.hdr key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:04.786963 29904 download_action.cpp:127] deal with download request finished! I1027 12:54:04.789415 29941 snapshot_manager.cpp:108] success to release snapshot path. [path='/data/storage/snapshot/20231027125404.1.180/'] I1027 12:54:04.789453 29941 agent_server.cpp:243] success to release_snapshot. snapshot_path=/data/storage/snapshot/20231027125404.1.180/, err_code=0 I1027 12:54:05.129667 29837 tablet_manager.cpp:886] begin to build all report tablets info I1027 12:54:05.130125 29837 tablet_manager.cpp:891] find expired transactions for 0 tablets I1027 12:54:05.130154 29836 data_dir.cpp:739] path: /data/storage total capacity: 1073215442944, available capacity: 1055869456384 I1027 12:54:05.133477 29835 task_worker_pool.cpp:1614] finish report TASK. master host: 192.168.10.100, port: 9020 I1027 12:54:05.135491 29814 task_worker_pool.cpp:483] get alter table task, signature: 3065358 I1027 12:54:05.135571 29814 schema_change.cpp:1407] begin to do request alter tablet: base_tablet_id=356929, base_schema_hash=1945210325, new_tablet_id=3065358, new_schema_hash=807381673, alter_version=367 I1027 12:54:05.135599 29814 schema_change.cpp:1462] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=356929.1945210325.b744aca517ce7e9e-949d395ebd78e9a2 new_tablet=3065358.807381673.114711b1c0014c81-8260117372f076b4 I1027 12:54:05.135658 29814 schema_change.cpp:1532] begin to remove all data from new tablet to prevent rewrite. new_tablet=3065358.807381673.114711b1c0014c81-8260117372f076b4 I1027 12:54:05.136073 29942 task_worker_pool.cpp:255] success to submit task. type=ALTER, signature=3065358, queue size=1 I1027 12:54:05.136161 29942 task_worker_pool.cpp:255] success to submit task. type=ALTER, signature=3065354, queue size=1 I1027 12:54:05.136202 29942 task_worker_pool.cpp:255] success to submit task. type=ALTER, signature=3065370, queue size=2 I1027 12:54:05.136260 29813 task_worker_pool.cpp:483] get alter table task, signature: 3065354 I1027 12:54:05.136308 29815 task_worker_pool.cpp:483] get alter table task, signature: 3065370 I1027 12:54:05.136260 29942 task_worker_pool.cpp:255] success to submit task. type=ALTER, signature=3065352, queue size=2 I1027 12:54:05.136390 29813 schema_change.cpp:1407] begin to do request alter tablet: base_tablet_id=356925, base_schema_hash=1945210325, new_tablet_id=3065354, new_schema_hash=807381673, alter_version=367 I1027 12:54:05.136420 29815 schema_change.cpp:1407] begin to do request alter tablet: base_tablet_id=356941, base_schema_hash=1945210325, new_tablet_id=3065370, new_schema_hash=807381673, alter_version=367 I1027 12:54:05.136426 29813 schema_change.cpp:1462] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=356925.1945210325.904422db3521d8d7-cf96a0d79a8a6f98 new_tablet=3065354.807381673.694a5f6803561eb8-a46979fa4eb076b2 I1027 12:54:05.136452 29815 schema_change.cpp:1462] finish to validate alter tablet request. begin to convert data from base tablet to new tablet base_tablet=356941.1945210325.434afb80b6ca9343-365ab7a81bb7c997 new_tablet=3065370.807381673.d6411e64569c6e29-055bc58e39199a97 I1027 12:54:05.136444 29942 task_worker_pool.cpp:255] success to submit task. type=ALTER, signature=3065364, queue size=2 I1027 12:54:05.136492 29813 schema_change.cpp:1532] begin to remove all data from new tablet to prevent rewrite. new_tablet=3065354.807381673.694a5f6803561eb8-a46979fa4eb076b2 I1027 12:54:05.136512 29815 schema_change.cpp:1532] begin to remove all data from new tablet to prevent rewrite. new_tablet=3065370.807381673.d6411e64569c6e29-055bc58e39199a97 I1027 12:54:05.138526 29813 schema_change.cpp:1841] begin to convert historical rowsets for new_tablet from base_tablet. base_tablet=356925.1945210325.904422db3521d8d7-cf96a0d79a8a6f98, new_tablet=3065354.807381673.694a5f6803561eb8-a46979fa4eb076b2 I1027 12:54:05.138594 29813 schema_change.cpp:1877] doing schema change directly for base_tablet 356925.1945210325.904422db3521d8d7-cf96a0d79a8a6f98 I1027 12:54:05.138669 29814 schema_change.cpp:1841] begin to convert historical rowsets for new_tablet from base_tablet. base_tablet=356929.1945210325.b744aca517ce7e9e-949d395ebd78e9a2, new_tablet=3065358.807381673.114711b1c0014c81-8260117372f076b4 I1027 12:54:05.138710 29815 schema_change.cpp:1841] begin to convert historical rowsets for new_tablet from base_tablet. base_tablet=356941.1945210325.434afb80b6ca9343-365ab7a81bb7c997, new_tablet=3065370.807381673.d6411e64569c6e29-055bc58e39199a97 I1027 12:54:05.138728 29814 schema_change.cpp:1877] doing schema change directly for base_tablet 356929.1945210325.b744aca517ce7e9e-949d395ebd78e9a2 I1027 12:54:05.138746 29815 schema_change.cpp:1877] doing schema change directly for base_tablet 356941.1945210325.434afb80b6ca9343-365ab7a81bb7c997 I1027 12:54:05.179544 29836 storage_engine.cpp:382] get root path info cost: 49 ms. tablet counter: 31481 I1027 12:54:05.180470 29836 task_worker_pool.cpp:1614] finish report DISK. master host: 192.168.10.100, port: 9020 I1027 12:54:05.223526 29837 tablet_manager.cpp:937] success to build all report tablets info. tablet_count=31481 I1027 12:54:05.466845 29837 task_worker_pool.cpp:1614] finish report TABLET. master host: 192.168.10.100, port: 9020 Aborted at 1698382445 (unix time) try "date -d @1698382445" if you are using GNU date SIGSEGV address not mapped to object (@0x55dc32303033) received by PID 29505 (TID 0x7f555f71c700) from PID 842018867; stack trace: I1027 12:54:05.795452 29941 snapshot_manager.cpp:303] receive a make snapshot request, request detail is TSnapshotRequest { 01: tablet_id (i64) = 1514471, 02: schema_hash (i32) = 1848757731, 05: timeout (i64) = 180, 09: preferred_snapshot_version (i32) = 4, }, snapshot_version is 4 I1027 12:54:05.797154 29941 snapshot_manager.cpp:88] success to make snapshot. path=['/data/storage/snapshot/20231027125405.2.180'] I1027 12:54:05.797186 29941 agent_server.cpp:223] success to make_snapshot. tablet_id=1514471, schema_hash=1848757731, snapshot_path: /data/storage/snapshot/20231027125405.2.180 I1027 12:54:05.799423 29902 download_action.cpp:107] accept one download request HttpRequest: method:0 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125405.2.180//1514471/1848757731/ raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125405.2.180//1514471/1848757731/ key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:05.799687 29902 download_action.cpp:127] deal with download request finished! I1027 12:54:05.800912 29934 download_action.cpp:107] accept one download request HttpRequest: method:4 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125405.2.180//1514471/1848757731/1514471.hdr raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125405.2.180//1514471/1848757731/1514471.hdr key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:05.801092 29934 download_action.cpp:127] deal with download request finished! I1027 12:54:05.802352 29902 download_action.cpp:107] accept one download request HttpRequest: method:0 uri:/api/_tablet/_download?token=74a069cd-eda3-42e8-b945-0dbe11437598&file=/data/storage/snapshot/20231027125405.2.180//1514471/1848757731/1514471.hdr raw_path:/api/_tablet/_download headers: key=Accept, value=/ key=Host, value=192.168.10.101:8040 params: key=file, value=/data/storage/snapshot/20231027125405.2.180//1514471/1848757731/1514471.hdr key=token, value=74a069cd-eda3-42e8-b945-0dbe11437598 I1027 12:54:05.802522 29902 download_action.cpp:127] deal with download request finished! I1027 12:54:05.804908 29941 snapshot_manager.cpp:108] success to release snapshot path. [path='/data/storage/snapshot/20231027125405.2.180/'] I1027 12:54:05.804945 29941 agent_server.cpp:243] success to release_snapshot. snapshot_path=/data/storage/snapshot/20231027125405.2.180/, err_code=0 0# 0x000055DC52C2E768 in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be 1# 0x00007F55D25F0400 in /lib64/libc.so.6 2# __memcmp_sse4_1 in /lib64/libc.so.6 3# doris::FieldTypeTraits<(doris::FieldType)13>::cmp(void const, void const) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be 4# doris::segment_v2::ZoneMapIndexWriter::add_values(void const*, unsigned long) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be 5# doris::segment_v2::ScalarColumnWriter::append_data_in_current_page(unsigned char const*, unsigned long) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be 6# doris::segment_v2::ScalarColumnWriter::append_data(unsigned char const*, unsigned long) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be 7# doris::segment_v2::ColumnWriter::append_nullable(unsigned char const, void const, unsigned long) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be 8# doris::Status doris::segment_v2::SegmentWriter::append_row(doris::RowCursor const&) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
9# doris::OLAPStatus doris::BetaRowsetWriter::_add_row(doris::RowCursor const&) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
10# doris::SchemaChangeDirectly::_write_row_block(doris::RowsetWriter , doris::RowBlock) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
11# doris::SchemaChangeDirectly::process(std::shared_ptr, doris::RowsetWriter , std::shared_ptr, std::shared_ptr) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
12# doris::SchemaChangeHandler::_convert_historical_rowsets(doris::SchemaChangeHandler::SchemaChangeParams const&) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
13# doris::SchemaChangeHandler::_do_process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
14# doris::SchemaChangeHandler::process_alter_tablet_v2(doris::TAlterTabletReqV2 const&) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
15# doris::EngineAlterTabletTask::execute() in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
16# doris::StorageEngine::execute_task(doris::EngineTask) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
17# doris::TaskWorkerPool::_alter_tablet(doris::TAgentTaskRequest const&, long, doris::TTaskType::type, doris::TFinishTaskRequest) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
18# doris::TaskWorkerPool::_alter_tablet_worker_thread_callback() in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
19# doris::ThreadPool::dispatch_thread() in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
20# doris::Thread::supervise_thread(void*) in /opt/doris/apache-doris-1.1.1-bin-x86/be/lib/doris_be
21# start_thread in /lib64/libpthread.so.0
22# clone in /lib64/libc.so.6
./bin/start_be.sh: 行 121: 29505 段错误 $LIMIT ${DORIS_HOME}/lib/doris_be "$@" 2>&1 < /dev/null
What You Expected?
no
How to Reproduce?
No response
Anything Else?
No response
Are you willing to submit PR?
Code of Conduct