search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
111 stars 38 forks source link

[Bug]: The data has been flushed before the backup, but there is much more data after the backup than before the backup. #265

Closed wutachiang closed 1 month ago

wutachiang commented 6 months ago

Current Behavior

milvus-backup version 0.4.2. The original cluster has multiple datasets, and only one of them was backed up. The data was flushed before the backup. The backup process is as follows: the mivlus-backup tool backs up the data of cluster A, and then synchronizes the S3 bucket backed up by cluster A to the new S3. bucket, and then used the mivus-backup tool to restore the backed up data, but found that the backed up data was much more than before, some even several times more. Source: Xnip2023-12-20_09-53-37 Destination: Xnip2023-12-20_09-54-28

Expected Behavior

No response

Steps To Reproduce

No response

Environment

milvus 2.2.11 cluster
milvus-backup 0.4.2
centos7

Anything else?

No response

zhuwenxing commented 6 months ago

There is an issue with delta log https://github.com/milvus-io/milvus/issues/29122 It means some deleted data was still in the backup and will be recovered. There is already a fix for it https://github.com/zilliztech/milvus-backup/pull/257. It was verified in Milvus v2.3.x, and I will take a look at Milvus v2.2.x.

wayblink commented 6 months ago

@wutachiang Hi, some known bugs may cause data redundancy. Do you have a lot of delete operations? Please try the latest v0.4.4. And We can see what's the problem.

wutachiang commented 6 months ago

@wayblink The number of collections restored after backup is incorrect, and they have a lot of deletion operations.

wutachiang commented 6 months ago

@zhuwenxing The previous backup process refers to the official document of mivlus "https://milvus.io/docs/milvus_backup_overview.md". That information is a bit old and needs to be updated.

wayblink commented 6 months ago

@wayblink The number of collections restored after backup is incorrect, and they have a lot of deletion operations.

Please upload your log and {backup_name}/meta folder if it is allowed. We can look into it.