search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.38k stars 3.22k forks source link

UnhealthyTable Too many mistakes #5790

Open pigdance opened 3 years ago

pigdance commented 3 years ago

version info Version: trunk Git: file:///root/incubator-doris-branch-0.14@Unknown Build Info: root@d4a76cc775da Build Time: Sun, 25 Apr 2021 03:01:26 UTC

mysql> show proc '/statistic'; +---------+----------------------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+ | DbId | DbName | TableNum | PartitionNum | IndexNum | TabletNum | ReplicaNum | UnhealthyTabletNum | InconsistentTabletNum | CloningTabletNum | +---------+----------------------------------------+----------+--------------+----------+-----------+------------+--------------------+-----------------------+------------------+ | 3382321 | default_cluster:db_test | 7 | 99 | 164 | 1538 | 3970 | 3 | 0 | 0 |

mysql> show proc '/statistic/3382321'; +-----------------------------+---------------------+----------------+ | UnhealthyTablets | InconsistentTablets | CloningTablets | +-----------------------------+---------------------+----------------+ | [4197672, 4197684, 4197688] | [] | [] | +-----------------------------+---------------------+----------------+ 1 row in set (0.00 sec)

mysql> SHOW PROC '/dbs/3382321/4196786/partitions/4196783/4196787/4197672'; +-----------+-----------+---------+---------------------+-------------------+-----------------------+------------------+----------------------+---------------------+------------+----------+----------+--------+-------+--------------+----------------------+---------------------------------------------------------------+-----------------------------------------------------------------------------------------+ | ReplicaId | BackendId | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | SchemaHash | DataSize | RowCount | State | IsBad | VersionCount | PathHash | MetaUrl | CompactionStatus | +-----------+-----------+---------+---------------------+-------------------+-----------------------+------------------+----------------------+---------------------+------------+----------+----------+--------+-------+--------------+----------------------+---------------------------------------------------------------+-----------------------------------------------------------------------------------------+ | 4197673 | 3397920 | 14149 | 3187970018723863348 | 14149 | 3187970018723863348 | 14440 | 5762713364219284511 | 2021-05-11 15:20:26 | 1604428979 | 30877005 | 269110 | NORMAL | false | 501 | 5720472665479996049 | http://x.x.x.221:18040/api/meta/header/4197672/1604428979 | http://x.x.x.221:18040/api/compaction/show?tablet_id=4197672&schema_hash=1604428979 | | 4197674 | 3399613 | 14440 | 5762713364219284511 | 14440 | 5762713364219284511 | -1 | 0 | NULL | 1604428979 | 30877005 | 269110 | NORMAL | false | 2 | 3532817995339609023 | http://x.x.x.228:18040/api/meta/header/4197672/1604428979 | http://x.x.x.228:18040/api/compaction/show?tablet_id=4197672&schema_hash=1604428979 | | 4197675 | 10234 | 14440 | 5762713364219284511 | 14440 | 5762713364219284511 | -1 | 0 | NULL | 1604428979 | 30877005 | 269110 | NORMAL | false | 2 | -5443766044805732626 | http://x.x.x.227:18040/api/meta/header/4197672/1604428979 | http://x.x.x.227:18040/api/compaction/show?tablet_id=4197672&schema_hash=1604428979 | +-----------+-----------+---------+---------------------+-------------------+-----------------------+------------------+----------------------+---------------------+------------+----------+----------+--------+-------+--------------+----------------------+---------------------------------------------------------------+-----------------------------------------------------------------------------------------+

be.info

I0511 15:37:18.250268 1048 task_worker_pool.cpp:879] get clone task. signature:4197672 I0511 15:37:18.254894 1048 engine_clone_task.cpp:287] success to make snapshot. ip=x.x.x.228, port=19060, tablet=4197672, schema_hash=1604428979, snapshot_path=/data/doris_install/storage/snapshot/20210511153718.489.180/, signature=4197672 I0511 15:37:18.260604 1048 engine_clone_task.cpp:475] clone begin to download file from: http://x.x.x.228:18040/api/_tablet/_download?token=af768ea6-0de7-4cd9-a152-4694dca8c609&file=/data/doris_install/storage/snapshot/20210511153718.489.180//4197672/1604428979/020000000031fcb22a4d5f17a61846c9ace022d0c9428abe_0.dat to: /data/doris_install/storage/data/1010/4197672/1604428979/clone/020000000031fcb22a4d5f17a61846c9ace022d0c9428abe_0.dat. size(B): 30877005, timeout(s): 603 I0511 15:37:18.525580 1048 engine_clone_task.cpp:475] clone begin to download file from: http://x.x.x.228:18040/api/_tablet/_download?token=af768ea6-0de7-4cd9-a152-4694dca8c609&file=/data/doris_install/storage/snapshot/20210511153718.489.180//4197672/1604428979/4197672.hdr to: /data/doris_install/storage/data/1010/4197672/1604428979/clone/4197672.hdr. size(B): 37977, timeout(s):300 I0511 15:37:18.527498 1048 engine_clone_task.cpp:507] succeed to copy tablet 4197672, total file size: 30914982 B, cost: 268 ms, rate: 115.354 MB/s I0511 15:37:18.529239 1048 beta_rowset.cpp:88] begin to remove files in rowset /data/doris_install/storage/data/1010/4197672/1604428979/clone//0200000000284c462a4d5f17a61846c9ace022d0c9428abe, version:0-1, tabletid:4197672 I0511 15:37:18.530251 1048 beta_rowset.cpp:88] begin to remove files in rowset /data/doris_install/storage/data/1010/4197672/1604428979/clone//020000000031fcb22a4d5f17a61846c9ace022d0c9428abe, version:2-14440, tabletid:4197672 I0511 15:37:18.530257 1048 beta_rowset.cpp:94] deleting /data/doris_install/storage/data/1010/4197672/1604428979/clone//020000000031fcb22a4d5f17a61846c9ace022d0c9428abe_0.dat W0511 15:37:18.530573 1048 beta_rowset.cpp:55] failed to open segment /data/doris_install/storage/data/1010/4197672/1604428979/clone//020000000031f3622a4d5f17a61846c9ace022d0c9428abe_0.dat under rowset /data/doris_install/storage/data/1010/4197672/1604428979/clone//020000000031f3622a4d5f17a61846c9ace022d0c9428abe : Not found: /data/doris_install/storage/data/1010/4197672/1604428979/clone//020000000031f3622a4d5f17a61846c9ace022d0c9428abe_0.dat: No such file or directory (error 2) W0511 15:37:18.530925 1048 engine_clone_task.cpp:328] fail to convert rowset ids, path=/data/doris_install/storage/data/1010/4197672/1604428979/clone/, tablet_id=4197672, schema_hash=1604428979, error=-3109 I0511 15:37:18.531420 1048 engine_clone_task.cpp:338] success to release snapshot, ip=x.x.x.228, port=19060, snapshot_path=/data/doris_install/storage/snapshot/20210511153718.489.180/ I0511 15:37:18.531436 1048 engine_clone_task.cpp:106] tablet exist with number of missing version: 159, try to incremental clone succeed: 0, signature: 4197672, tablet id: 4197672, schema hash: 1604428979, clone version: 14440, download snapshot: -1

pigdance commented 3 years ago

Data import failed due to unhealthtable Please also help analyze the reasons .thanks

org.apache.kafka.connect.errors.RetriableException: Stream load failed, statusCode=200 load result= loadUrl:http://x.x.x.225:18030/api/db_1/table_1/_stream_load { "TxnId": 14618738, "Label": "d692c470-c79a-4399-a894-a4ef27ca89b2", "Status": "Fail", "Message": "already stopped, skip waiting for close. cancelled/!eos: : 1/0", "NumberTotalRows": 0, "NumberLoadedRows": 0, "NumberFilteredRows": 0, "NumberUnselectedRows": 0, "LoadBytes": 51776, "LoadTimeMs": 266, "BeginTxnTimeMs": 0, "StreamLoadPutTimeMs": 3, "ReadDataTimeMs": 0, "WriteDataTimeMs": 259, "CommitAndPublishTimeMs": 0 }