search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
31.04k stars 2.95k forks source link

[Bug]:collection remain in loading state if lost delta_logs #36053

Open alienzyl opened 2 months ago

alienzyl commented 2 months ago

Is there an existing issue for this?

Environment

- Milvus version: 2.2.4
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

I lost two delta_logs, which has caused two of my collections to remain in a loading state and unable to complete. How can I recover from this situation? image

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

[2024/09/06 07:23:42.557 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.609 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.612 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.663 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.666 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.718 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.719 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.771 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.774 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.826 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.828 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.889 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.891 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:42.945 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:42.946 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.001 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.001 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.055 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.056 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.110 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.110 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.164 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.164 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.218 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.219 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.273 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.273 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.327 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.327 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.382 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.382 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.436 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.436 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.490 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.490 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.543 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.544 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.597 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.599 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.651 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.653 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.706 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.707 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.761 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.762 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.817 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.817 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.871 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.871 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.925 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.926 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:43.981 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:43.984 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:44.039 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:44.047 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:44.094 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:44.101 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:44.147 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:44.156 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:44.202 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:44.210 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:44.256 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"] [2024/09/06 07:23:44.265 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647908746077/443707647908746130/451442729517055455/451442729517121919]"] [2024/09/06 07:23:44.310 +00:00] [WARN] [datanode/binlog_io.go:85] ["downloading failed, retry in 50ms"] [paths="[file/delta_log/443707647914845261/443707647914845280/451442729516966119/451442729517121114]"]

Anything else?

No response

jaime0815 commented 2 months ago

@alienzyl Data has been lost, and fixing it isn't easy. I recommend rebuilding the cluster if it's a testing environment, or you may need to remove the binlog metadata of the two delta logs to bypass deletion.

yanliang567 commented 2 months ago

/assign @alienzyl /unassign

xiaofan-luan commented 2 months ago
  1. what's the reason you lost two delta logs?
  2. you can use birdwatcher to remove the meta info so it won't be loaded
stale[bot] commented 2 weeks ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.