milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.36k stars 2.91k forks source link

[Bug]: milvus query is hang #35752

Open liuzh-sa opened 2 months ago

liuzh-sa commented 2 months ago

Is there an existing issue for this?

Environment

- Milvus version: 2.4.4
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

query hang

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

milvus-log.tar.gz

Anything else?

No response

xiaofan-luan commented 2 months ago

file/delta_log/451293960015077677/451293960015077678/451293960016289982/451792976385935885

there seems to be some delta log missing..

you probably need to check the log on how many files are missing. If there are only 1-2 files missing maybe simply use birdwatcher to remove the meta.

liuzh-sa commented 2 months ago

file/delta_log/451293960015077677/451293960015077678/451293960016289982/451792976385935885

there seems to be some delta log missing..

you probably need to check the log on how many files are missing. If there are only 1-2 files missing maybe simply use birdwatcher to remove the meta.

How to delete?

congqixia commented 2 months ago

@liuzh-sa

  1. how to check how many files is missing birdwatcher has a command to scan all missing file verify-segment --collection [colleciton_id]

  2. use remove binlog to remove error binlog if found

liuzh-sa commented 2 months ago
  1. verify-segment --collection

image Do we need to delete all of these?

yanliang567 commented 2 months ago

/assign @congqixia /unassign

xiaofan-luan commented 2 months ago
  1. verify-segment --collection

image Do we need to delete all of these?

seems that you are missing a lot of data.

Could you explain what operation you did to this cluster?

liuzh-sa commented 2 months ago
  1. verify-segment --collection

image Do we need to delete all of these?

seems that you are missing a lot of data.

Could you explain what operation you did to this cluster?

We imported data into the cluster. Before the import started, the cluster became unreadable. Then we restarted the cluster and that was it.

liuzh-sa commented 2 months ago

小龙 understands the situation of our cluster.

stale[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.