search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
135 stars 50 forks source link

[Bug]: most collections are not being restored #430

Closed lindner-tj closed 2 months ago

lindner-tj commented 2 months ago

Current Behavior

Hello All, I'm facing some issues setting up a backup routine for Milvus 2.4.9.

command: ./milvus-backup restore -n my_backup -s _recover

outcome: out of my 6 DBs with 1-3 collections each only 2 collections were restored in total. But every database is at least being created (but not filled) when restoring a completely empty system.

logs:

root@28d47480526e:/opt/bin# ./milvus-backup restore -n my_backup -s _recover
0.4.23 (Built on 2024-09-12T07:00:30Z from Git SHA 54c93d608cec9e7882ccee3b6fe47e418d8f3642)
[2024/09/20 06:27:38.276 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/09/20 06:27:38.276 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/09/20 06:27:38.277 +00:00] [INFO] [cmd/restore.go:47] ["restore cmd input args"] [args="[]"]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_impl_restore_backup.go:29] ["receive RestoreBackupRequest"] [requestId=6e743b86-7719-11ef-8ea4-0242ac160007] [backupName=my_backup] [onlyMeta=false] [restoreIndex=false] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=_recover] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [skipDiskQuotaCheck=false]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_context.go:117] ["{Base:0xc00014a2c0 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://milvus-standalone-temp:9091}"]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_context.go:118] ["{Base:0xc00014a2c0 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_context.go:308] ["receive GetBackupRequest"] [requestId=6e7442e7-7719-11ef-8ea4-0242ac160007] [backupName=my_backup] [backupId=] [bucketName=] [path=]
[2024/09/20 06:27:38.277 +00:00] [DEBUG] [core/backup_context.go:192] ["create backup storage client"] [address=milvus-backup-storage:9000] [bucket=a-bucket] [backupBucket=a-bucket]
[2024/09/20 06:27:38.280 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/09/20 06:27:38.287 +00:00] [DEBUG] [core/backup_context.go:378] ["finish GetBackupRequest"] [requestId=6e7442e7-7719-11ef-8ea4-0242ac160007] [backupName=my_backup] [backupId=] [bucketName=] [path=]
[2024/09/20 06:27:38.287 +00:00] [INFO] [core/backup_impl_restore_backup.go:199] ["Collections to restore"] [collection_num=8]
[2024/09/20 06:27:38.287 +00:00] [DEBUG] [core/backup_context.go:68] ["Start Milvus client"] [endpoint=milvus-standalone-temp:19530]
[2024/09/20 06:27:38.323 +00:00] [INFO] [core/backup_impl_restore_backup.go:354] ["Start collection level restore pool"] [parallelism=2]
[2024/09/20 06:27:38.324 +00:00] [INFO] [core/backup_impl_restore_backup.go:358] ["executeRestoreBackupTask start"] [backup_name=my_backup] [backupBucketName=a-bucket] [backupPath=files/my_backup]
[2024/09/20 06:27:38.324 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [backupBucketName=a-bucket] [backupPath=files/my_backup]
[2024/09/20 06:27:38.324 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["collection schema"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"id\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"chunk_index\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Chunk index\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":102,\"Name\":\"headline\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Headline\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"200\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":103,\"Name\":\"text\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Text chunk\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"65535\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":104,\"Name\":\"embedding\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Embedding vectors\",\"DataType\":101,\"TypeParams\":{\"dim\":\"3072\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0}]"]
[2024/09/20 06:27:38.326 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [backupBucketName=a-bucket] [backupPath=files/my_backup]
[2024/09/20 06:27:38.326 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["collection schema"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"id\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"text\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Text chunk\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"65535\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":102,\"Name\":\"embedding\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Embedding vectors\",\"DataType\":101,\"TypeParams\":{\"dim\":\"384\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0}]"]
[2024/09/20 06:27:38.356 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["create collection"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [hasPartitionKey=false]
[2024/09/20 06:27:38.357 +00:00] [DEBUG] [core/backup_context.go:161] ["create milvus storage client"] [address=milvus-minio-temp:9000] [bucket=a-bucket] [backupBucket=a-bucket]
[2024/09/20 06:27:38.359 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/09/20 06:27:38.359 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore partition"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [partition=_default]
[2024/09/20 06:27:38.379 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["create collection"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [hasPartitionKey=false]
[2024/09/20 06:27:38.379 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore partition"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [partition=_default]
[2024/09/20 06:27:38.382 +00:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=a-bucket] [backupPath=files/my_backup] [partitionID=452629157848001990] [groupId=452629157848002002]
[2024/09/20 06:27:38.384 +00:00] [INFO] [core/backup_impl_restore_backup.go:760] ["execute bulk insert"] [db=some_db] [collection=website_text_recover] [partition=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/,]"] [endTime=452655854222835713]
[2024/09/20 06:27:38.385 +00:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=a-bucket] [backupPath=files/my_backup] [partitionID=452629157848001823] [groupId=452629157848001835]
[2024/09/20 06:27:38.387 +00:00] [INFO] [core/backup_impl_restore_backup.go:760] ["execute bulk insert"] [db=internal_chatbot] [collection=bz_data_recover] [partition=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001822/452629157848001823/452629157848001835/,]"] [endTime=452655854222835713]
[2024/09/20 06:27:38.395 +00:00] [ERROR] [core/backup_impl_restore_backup.go:782] ["fail to bulk insert"] [db=some_db] [collectionName=website_text_recover] [partitionName=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/,]"] [error="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:782\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.396 +00:00] [ERROR] [core/backup_impl_restore_backup.go:685] ["fail to bulk insert to partition"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [partition=_default] [error="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.396 +00:00] [ERROR] [core/backup_impl_restore_backup.go:371] ["executeRestoreCollectionTask failed"] [TargetDBName=some_db] [TargetCollectionName=website_text_recover] [error="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:371\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.396 +00:00] [ERROR] [core/backup_impl_restore_backup.go:782] ["fail to bulk insert"] [db=internal_chatbot] [collectionName=bz_data_recover] [partitionName=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001822/452629157848001823/452629157848001835/,]"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:782\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.397 +00:00] [ERROR] [core/backup_impl_restore_backup.go:685] ["fail to bulk insert to partition"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [partition=_default] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.397 +00:00] [ERROR] [core/backup_impl_restore_backup.go:371] ["executeRestoreCollectionTask failed"] [TargetDBName=internal_chatbot] [TargetCollectionName=bz_data_recover] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:371\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.397 +00:00] [ERROR] [core/backup_impl_restore_backup.go:335] ["execute restore collection fail"] [backupId=ca75a429-7679-11ef-b094-0242ac160006] [error="workerpool: execute job no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:335\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"]
workerpool: execute job no binlog to import, input=[paths:"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/" paths:"" ]: invalid parameter
duration:0 s

Expected Behavior

all collections to be present in my DBs

Steps To Reproduce

1. backing up (works) with `./milvus-backup create -n my_backup`
2. restoring doesn't work, command: `./milvus-backup restore -n my_backup -s _recover`

I also tried to manually copy the data from the backup a-bucket to the milvus a-bucket before running the restore, but same result.

Environment

- milvus 2.4.9 via docker compose
- milvus-backup 0.4.23 running in a bitnami/pymilvus:2.4.6-debian-12-r2 container
- version of Milvus minio = Version of mivlus-backup minio = minio/minio:RELEASE.2023-03-20T20-16-18Z
- names: I want to backup and restore "milvus-minio-temp" to and from "milvus-backup-storage"

Anything else?

No response

lindner-tj commented 2 months ago

Update: Restoring using another milvus-instance and cross-site backup works flawlessly

wayblink commented 2 months ago

@lindner-tj Hi, as you described, restore succeeded partially？We haven't received a similar issue. Not sure what happened. Have you checked the path in the error message? Do they exist in storage? If your backup data is small enough and safe to share, you can send it to me.

lindner-tj commented 2 months ago

@wayblink Unfortunately there is too much confidential stuff in our DBs, thank you for the offer, though. I am going to close this issue, since the cross-site backup and restore works fine. I only found out that cross-site works after I had already opened this issue.