[Bug]: It seems that the indexnode is not working and the number of unissued index tasks keeps increasing

[ThreadDao]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: master-20240918-a7dcc50f
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

test steps

1. create a collection with 1 shard and 16 num_partitions
2. insert 30m-128d data
3. flush -> index -> load
4. concurrent requests: search + upsert + flush
5. dataCoord shows that collection has 14.8k L1 segments, indexNode was not working and unissued tasks keeps increasing
   • segment num and index task
   • indexNode cpu usage
   • the mixcoord CPU usage is relatively high By the way, proxy and mixcoord restart twice due to lost conmnection from etcd

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/d026185e-699d-49eb-b4a5-8fc54af0ddae?nodeId=level-zero-stable-master-1726671600-2235253561

Milvus Log

level-master-pa-key-op-44-3730-etcd-0                             1/1     Running       0                36h     10.104.19.59    4am-node28   <none>           <none>
level-master-pa-key-op-44-3730-etcd-1                             1/1     Running       0                36h     10.104.30.2     4am-node38   <none>           <none>
level-master-pa-key-op-44-3730-etcd-2                             1/1     Running       0                36h     10.104.27.38    4am-node31   <none>           <none>
level-master-pa-key-op-44-3730-milvus-datanode-6c856c7db-6fgxq    1/1     Running       1 (36h ago)      36h     10.104.1.207    4am-node10   <none>           <none>
level-master-pa-key-op-44-3730-milvus-indexnode-7d8bbf4968cvbrn   1/1     Running       0                36h     10.104.20.245   4am-node22   <none>           <none>
level-master-pa-key-op-44-3730-milvus-indexnode-7d8bbf4968phzzw   1/1     Running       0                36h     10.104.33.149   4am-node36   <none>           <none>
level-master-pa-key-op-44-3730-milvus-mixcoord-5b44b997c8-84nxd   1/1     Running       2 (31h ago)      36h     10.104.13.102   4am-node16   <none>           <none>
level-master-pa-key-op-44-3730-milvus-proxy-5dbd8c8679-kzzp9      1/1     Running       2 (31h ago)      36h     10.104.1.206    4am-node10   <none>           <none>
level-master-pa-key-op-44-3730-milvus-querynode-0-78577d89d5d6h   1/1     Running       0                36h     10.104.17.15    4am-node23   <none>           <none>
level-master-pa-key-op-44-3730-milvus-querynode-0-78577d89pbv58   1/1     Running       5 (25h ago)      36h     10.104.18.229   4am-node25   <none>           <none>
level-master-pa-key-op-44-3730-milvus-querynode-0-78577d89r45sd   1/1     Running       1 (29h ago)      36h     10.104.20.247   4am-node22   <none>           <none>
level-master-pa-key-op-44-3730-milvus-querynode-0-78577d89s9vcz   1/1     Running       1 (25h ago)      36h     10.104.32.190   4am-node39   <none>           <none>
level-master-pa-key-op-44-3730-minio-0                            1/1     Running       0                36h     10.104.27.42    4am-node31   <none>           <none>
level-master-pa-key-op-44-3730-minio-1                            1/1     Running       0                36h     10.104.34.105   4am-node37   <none>           <none>
level-master-pa-key-op-44-3730-minio-2                            1/1     Running       0                36h     10.104.25.243   4am-node30   <none>           <none>
level-master-pa-key-op-44-3730-minio-3                            1/1     Running       0                36h     10.104.19.62    4am-node28   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-bookie-0                    1/1     Running       0                36h     10.104.34.106   4am-node37   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-bookie-1                    1/1     Running       0                36h     10.104.27.43    4am-node31   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-bookie-2                    1/1     Running       0                36h     10.104.15.93    4am-node20   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-broker-0                    1/1     Running       0                36h     10.104.1.197    4am-node10   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-proxy-0                     1/1     Running       0                36h     10.104.4.198    4am-node11   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-recovery-0                  1/1     Running       0                36h     10.104.25.241   4am-node30   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-zookeeper-0                 1/1     Running       0                36h     10.104.19.63    4am-node28   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-zookeeper-1                 1/1     Running       0                36h     10.104.34.113   4am-node37   <none>           <none>
level-master-pa-key-op-44-3730-pulsar-zookeeper-2                 1/1     Running       0                36h     10.104.16.68    4am-node21   <none>           <none>

Anything else?

No response

[xiaocai2333]

Currently, a stats task is performed before building the index for segments. Due to the large number of segments, it appears on the monitoring system that the number of unissued index tasks is slowly increasing. In fact, this is because the stats tasks are completing slowly. The reason for the slow completion of the stats tasks is that the connection to MinIO/S3 on the index node is taking too long (more than 10 seconds), causing the datacoord to think that the task assignment has failed. However, the index node has already cached the task. When datacoord reassigns the task, it does not first clear the cache on the index node, resulting in the index node reporting that the task already exists during reassignment. This blocks the task's progress, which explains why tasks execute quickly after the index node is restarted. And pull request #36371 will fix it.