add index blocked with error "not found instance” after injection pd leader io delay

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

1、enable tidb_enable_dist_task and global sort 2、run sysbench 3、add index for sbtest1 and sbtest2 concurrently 4、inject pd leader io delay 5s

operator logs： the status of ddl job is not synced after 1h0m0s (now: 2024-05-14 21:05:56, jobId: 546, job type: add index / ingest cloud /, state: running) operatorLogs: [2024-05-14 20:05:50] ###### start adding index ALTER TABLE sbtest2 ADD INDEX index_test_1715688350713(c) [2024-05-14 20:05:50] ###### wait for ddl job finish [2024-05-14 21:05:56] ###### wait for ddl job finish timeout(1h0m0s) select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE sbtest2 ADD INDEX index_test_1715688350713(c)' jobId: 546, job type: add index / ingest cloud /, state: running

2. What did you expect to see? (Required)

add index can success after fault recover

3. What did you see instead (Required)

add index blocked with error "not found instance” [2024/05/14 23:31:22.997 +08:00] [INFO] [scheduler.go:264] ["schedule task meet err, reschedule it"] [task-id=78] [task-type=backfill] [allocated-slots=true] [error="not found instance"]

4. What is your TiDB version? (Required)

./tidb-server -V Release Version: v8.1.0-alpha Edition: Community Git Commit Hash: 48a362641a67343fddf6ddcf0f59bcaeb17be5d5 Git Branch: heads/refs/tags/v8.1.0-alpha UTC Build Time: 2024-05-14 07:09:37 GoVersion: go1.21.6 Race Enabled: false Check Table Before Drop: false Store: unistore 2024-05-14T15:24:02.701+0800

pingcap / tidb