search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.72k stars 3.28k forks source link

[Bug] (engine_clone) version incomplete replica can't be repaired #16424

Open zhengshengjun opened 1 year ago

zhengshengjun commented 1 year ago

Search before asking

Version

1.2

What's Wrong?

We got a tablet replica that can't be repaired by engine clone task for a long time.It seems that engine clone task is most likely to fail if the version-missing repairing replica accept continuous data writes. We'v suffered from this many times, for it causes collocate group unstable and data loss risks.

What You Expected?

engine clone task work nicely for version missing replicas

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

Code of Conduct

zhengshengjun commented 1 year ago

FE Log:

2023-02-04 09:12:46,169 INFO (tablet scheduler|62) [TabletScheduler.schedulePendingTablets():457] add clone task to agent task queue: tablet id: 338714, replica id: 542836, schema hash: 156405523, storageMedium: HDD, visible version(hash): 7295011, src backend: xxxx:xxxx:x:xxx::57, src path hash: 4248943622730157877, src backend: xxxx:xxxx:x:xxx::57, src path hash: 4248943622730157877, dest backend: 10003, dest path hash: 9102637428361925408
2023-02-04 10:50:00,686 INFO (tablet scheduler|62) [TabletScheduler.removeTabletCtx():1425] remove the tablet 338714. because: timeout
2023-02-04 10:50:06,708 INFO (tablet scheduler|62) [TabletScheduler.schedulePendingTablets():457] add clone task to agent task queue: tablet id: 338714, replica id: 542836, schema hash: 156405523, storageMedium: HDD, visible version(hash): 7297643, src backend: xxxx:xxxx:x:xxx::40, src path hash: -2641615873456264123, src backend: xxxx:xxxx:x:xxx::40, src path hash: -2641615873456264123, dest backend: 10003, dest path hash: 9102637428361925408
2023-02-04 12:27:26,814 INFO (tablet scheduler|62) [TabletScheduler.removeTabletCtx():1425] remove the tablet 338714. because: timeout

BE Log:

0204 09:12:46.169904 2756190 engine_clone_task.cpp:105] clone to existed tablet. missed_versions_size=202, allow_incremental_clone=0, signature=338714, tablet_id=338714, committed_version=7295011
I0204 09:12:46.174667 2756190 engine_clone_task.cpp:228] successfully make snapshot in remote BE|host=xxxx:xxxx:x:443::57|port=9060|tablet=338714|snapshot_path=/data08/doris_data/xxx/be.HDD/snapshot/20230204091246.917.5834/|signature=338714|missed_versions=[7256165-7256165][7256166-7256166][7256167-7256167][7256168-7256168][7256169-7256169][7256170-7256170] ...... [7256365-7256365][7256366-7256366]
I0204 09:12:46.175863 2756190 engine_clone_task.cpp:408] clone begin to download file from: http://[xxxx:xxxx:x:xxx::57]:8040/api/_tablet/_download?token=ae7e81ff-d3f4-490f-8608-9a3096f00d3c&file=/data08/doris_data/aurora_data_hl/be.HDD/snapshot/20230204091246.917.5834//338714/156405523/020000000061024c0349f83cebdd79b9dcaa3797c7a8cc90_22.dat to: /data25/doris_data/xxx/be.SSD/data/134/338714/156405523/clone/020000000061024c0349f83cebdd79b9dcaa3797c7a8cc90_22.dat. size(B): 236173615, timeout(s): 4612
...
...
I0204 09:14:49.279039 2756190 engine_clone_task.cpp:446] succeed to copy tablet 338714, total file size: 6055467667 B, cost: 123103 ms, rate: 49.1903 MB/s
I0204 09:14:49.381126 2756190 engine_clone_task.cpp:573] begin to finish full clone. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, cloned_max_version=[7295003-7295012]
I0204 09:14:49.381153 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[0-7247166]
I0204 09:14:49.381157 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[0-7247166]
I0204 09:14:49.381160 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7247167-7256164]
I0204 09:14:49.381161 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7247167-7256164]
I0204 09:14:49.381163 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7256367-7265993]
I0204 09:14:49.381165 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7256367-7265993]
I0204 09:14:49.381166 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7265994-7277617]
I0204 09:14:49.381167 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7265994-7277617]
I0204 09:14:49.381170 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7277618-7290348]
I0204 09:14:49.381171 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7277618-7290348]
I0204 09:14:49.381173 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7290349-7294744]
I0204 09:14:49.381175 2756190 engine_clone_task.cpp:625] version exist in local tablet, no need to clone. delete it from clone tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, version='[7290349-7294744]
I0204 09:14:49.381197 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7294745-7295001]
I0204 09:14:49.381198 2756190 engine_clone_task.cpp:630] version not exist in local tablet. it will be replaced by other version. delete it from local tablet. tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8,, version=[7294745-7295001]
I0204 09:14:49.381201 2756190 engine_clone_task.cpp:594] check local delta when full clone.tablet=338714.156405523.4340c7b9f9a22030-e9049a6a3d0397b8, local_version=[7295002-7295066]
W0204 09:14:52.411546 2756190 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7295003-7295012], local_version=[7295002-7295066]

and the repeated err message in be is:

W0204 09:14:52.411546 2756190 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7295003-7295012], local_version=[7295002-7295066]
W0204 10:52:15.685870 2756189 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7297643-7297643], local_version=[7297537-7297677]
W0204 12:29:55.985198 2756189 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7300464-7300464], local_version=[7300336-7300467]
...
W0204 22:17:34.976135 2756189 task_worker_pool.cpp:925] failed to clone tablet|signature=338714|tablet_id=338714|error=Internal error: version cross src latest. cloned_max_version=[7316482-7316484], local_version=[7316296-7316502]
zhengshengjun commented 1 year ago

Consider the following case:

  1. if local version is : 0-0, 1-1, 2-10, 12-12, the missed version is 11
  2. start to snap shot and download from src be, the cloned max version is 13-13
  3. local be accept data writes during clone and triggered compaction, now the version is 0-0, 1-1, 2-10, 12-15
  4. the clone down load finished
  5. now 12-15 , report error: version cross src latest