search

zilliztech / milvus-backup

Backup and restore tool for Milvus
Apache License 2.0
126 stars 45 forks source link

[Bug]: most collections are not being restored #430

Open lindner-tj opened 3 hours ago

lindner-tj commented 3 hours ago

Current Behavior

Hello All, I'm facing some issues setting up a backup routine for Milvus 2.4.9.

command: ./milvus-backup restore -n my_backup -s _recover

outcome: out of my 6 DBs with 1-3 collections each only 2 collections were restored in total. But every database is at least being created (but not filled) when restoring a completely empty system.

logs:

root@28d47480526e:/opt/bin# ./milvus-backup restore -n my_backup -s _recover
0.4.23 (Built on 2024-09-12T07:00:30Z from Git SHA 54c93d608cec9e7882ccee3b6fe47e418d8f3642)
[2024/09/20 06:27:38.276 +00:00] [INFO] [logutil/logutil.go:165] ["Log directory"] [configDir=]
[2024/09/20 06:27:38.276 +00:00] [INFO] [logutil/logutil.go:166] ["Set log file to "] [path=logs/backup.log]
[2024/09/20 06:27:38.277 +00:00] [INFO] [cmd/restore.go:47] ["restore cmd input args"] [args="[]"]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_impl_restore_backup.go:29] ["receive RestoreBackupRequest"] [requestId=6e743b86-7719-11ef-8ea4-0242ac160007] [backupName=my_backup] [onlyMeta=false] [restoreIndex=false] [useAutoIndex=false] [dropExistCollection=false] [dropExistIndex=false] [skipCreateCollection=false] [collections="[]"] [CollectionSuffix=_recover] [CollectionRenames={}] [async=false] [bucketName=] [path=] [databaseCollections=] [skipDiskQuotaCheck=false]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_context.go:117] ["{Base:0xc00014a2c0 MaxSegmentGroupSize:2147483648 BackupCollectionParallelism:4 BackupCopyDataParallelism:128 RestoreParallelism:2 KeepTempFiles:false GcPauseEnable:true GcPauseSeconds:7200 GcPauseAddress:http://milvus-standalone-temp:9091}"]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_context.go:118] ["{Base:0xc00014a2c0 Enabled:true DebugMode:false SimpleResponse:true}"]
[2024/09/20 06:27:38.277 +00:00] [INFO] [core/backup_context.go:308] ["receive GetBackupRequest"] [requestId=6e7442e7-7719-11ef-8ea4-0242ac160007] [backupName=my_backup] [backupId=] [bucketName=] [path=]
[2024/09/20 06:27:38.277 +00:00] [DEBUG] [core/backup_context.go:192] ["create backup storage client"] [address=milvus-backup-storage:9000] [bucket=a-bucket] [backupBucket=a-bucket]
[2024/09/20 06:27:38.280 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/09/20 06:27:38.287 +00:00] [DEBUG] [core/backup_context.go:378] ["finish GetBackupRequest"] [requestId=6e7442e7-7719-11ef-8ea4-0242ac160007] [backupName=my_backup] [backupId=] [bucketName=] [path=]
[2024/09/20 06:27:38.287 +00:00] [INFO] [core/backup_impl_restore_backup.go:199] ["Collections to restore"] [collection_num=8]
[2024/09/20 06:27:38.287 +00:00] [DEBUG] [core/backup_context.go:68] ["Start Milvus client"] [endpoint=milvus-standalone-temp:19530]
[2024/09/20 06:27:38.323 +00:00] [INFO] [core/backup_impl_restore_backup.go:354] ["Start collection level restore pool"] [parallelism=2]
[2024/09/20 06:27:38.324 +00:00] [INFO] [core/backup_impl_restore_backup.go:358] ["executeRestoreBackupTask start"] [backup_name=my_backup] [backupBucketName=a-bucket] [backupPath=files/my_backup]
[2024/09/20 06:27:38.324 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [backupBucketName=a-bucket] [backupPath=files/my_backup]
[2024/09/20 06:27:38.324 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["collection schema"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"id\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"chunk_index\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Chunk index\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":102,\"Name\":\"headline\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Headline\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"200\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":103,\"Name\":\"text\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Text chunk\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"65535\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":104,\"Name\":\"embedding\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Embedding vectors\",\"DataType\":101,\"TypeParams\":{\"dim\":\"3072\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0}]"]
[2024/09/20 06:27:38.326 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [backupBucketName=a-bucket] [backupPath=files/my_backup]
[2024/09/20 06:27:38.326 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["collection schema"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [fields="[{\"ID\":100,\"Name\":\"id\",\"PrimaryKey\":true,\"AutoID\":true,\"Description\":\"id\",\"DataType\":5,\"TypeParams\":{},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":101,\"Name\":\"text\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Text chunk\",\"DataType\":21,\"TypeParams\":{\"max_length\":\"65535\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0},{\"ID\":102,\"Name\":\"embedding\",\"PrimaryKey\":false,\"AutoID\":false,\"Description\":\"Embedding vectors\",\"DataType\":101,\"TypeParams\":{\"dim\":\"384\"},\"IndexParams\":{},\"IsDynamic\":false,\"IsPartitionKey\":false,\"IsClusteringKey\":false,\"ElementType\":0}]"]
[2024/09/20 06:27:38.356 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["create collection"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [hasPartitionKey=false]
[2024/09/20 06:27:38.357 +00:00] [DEBUG] [core/backup_context.go:161] ["create milvus storage client"] [address=milvus-minio-temp:9000] [bucket=a-bucket] [backupBucket=a-bucket]
[2024/09/20 06:27:38.359 +00:00] [INFO] [storage/minio_chunk_manager.go:144] ["minio chunk manager init success."] [bucketname=a-bucket] [root=files]
[2024/09/20 06:27:38.359 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore partition"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [partition=_default]
[2024/09/20 06:27:38.379 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["create collection"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [hasPartitionKey=false]
[2024/09/20 06:27:38.379 +00:00] [INFO] [core/backup_impl_restore_backup.go:369] ["start restore partition"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [partition=_default]
[2024/09/20 06:27:38.382 +00:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=a-bucket] [backupPath=files/my_backup] [partitionID=452629157848001990] [groupId=452629157848002002]
[2024/09/20 06:27:38.384 +00:00] [INFO] [core/backup_impl_restore_backup.go:760] ["execute bulk insert"] [db=some_db] [collection=website_text_recover] [partition=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/,]"] [endTime=452655854222835713]
[2024/09/20 06:27:38.385 +00:00] [INFO] [core/backup_impl_restore_backup.go:867] [getBackupPartitionPaths] [bucketName=a-bucket] [backupPath=files/my_backup] [partitionID=452629157848001823] [groupId=452629157848001835]
[2024/09/20 06:27:38.387 +00:00] [INFO] [core/backup_impl_restore_backup.go:760] ["execute bulk insert"] [db=internal_chatbot] [collection=bz_data_recover] [partition=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001822/452629157848001823/452629157848001835/,]"] [endTime=452655854222835713]
[2024/09/20 06:27:38.395 +00:00] [ERROR] [core/backup_impl_restore_backup.go:782] ["fail to bulk insert"] [db=some_db] [collectionName=website_text_recover] [partitionName=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/,]"] [error="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:782\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.396 +00:00] [ERROR] [core/backup_impl_restore_backup.go:685] ["fail to bulk insert to partition"] [backup_db_name=some_db] [backup_collection_name=website_text] [target_db_name=some_db] [target_collection_name=website_text_recover] [skipDiskQuotaCheck=false] [partition=_default] [error="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.396 +00:00] [ERROR] [core/backup_impl_restore_backup.go:371] ["executeRestoreCollectionTask failed"] [TargetDBName=some_db] [TargetCollectionName=website_text_recover] [error="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [errorVerbose="no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus-sdk-go/v2/client.handleRespStatus\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/collection.go:39\n  | github.com/milvus-io/milvus-sdk-go/v2/client.(*GrpcClient).BulkInsert\n  | \t/home/runner/go/pkg/mod/github.com/milvus-io/milvus-sdk-go/v2@v2.4.2-0.20240821030256-0c339b63d265/client/insert.go:443\n  | github.com/zilliztech/milvus-backup/core.(*MilvusClient).BulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/milvus_sdk_wrapper.go:124\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:778\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\n  | github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n  | \t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\n  | github.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n  | \t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\n  | golang.org/x/sync/errgroup.(*Group).Go.func1\n  | \t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75\n  | runtime.goexit\n  | \t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/asm_amd64.s:1571\nWraps: (2) no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter\nError types: (1) *withstack.withStack (2) *errutil.leafError"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:371\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.396 +00:00] [ERROR] [core/backup_impl_restore_backup.go:782] ["fail to bulk insert"] [db=internal_chatbot] [collectionName=bz_data_recover] [partitionName=_default] [files="[files/my_backup/binlogs/insert_log/452629157848001822/452629157848001823/452629157848001835/,]"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeBulkInsert\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:782\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:605\ngithub.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.397 +00:00] [ERROR] [core/backup_impl_restore_backup.go:685] ["fail to bulk insert to partition"] [backup_db_name=internal_chatbot] [backup_collection_name=bz_data] [target_db_name=internal_chatbot] [target_collection_name=bz_data_recover] [skipDiskQuotaCheck=false] [partition=_default] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func7\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:685\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.397 +00:00] [ERROR] [core/backup_impl_restore_backup.go:371] ["executeRestoreCollectionTask failed"] [TargetDBName=internal_chatbot] [TargetCollectionName=bz_data_recover] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreBackupTask.func1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:371\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpool.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.3.0/errgroup/errgroup.go:75"]
[2024/09/20 06:27:38.397 +00:00] [ERROR] [core/backup_impl_restore_backup.go:335] ["execute restore collection fail"] [backupId=ca75a429-7679-11ef-b094-0242ac160006] [error="workerpool: execute job no binlog to import, input=[paths:\"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/\" paths:\"\" ]: invalid parameter"] [stack="github.com/zilliztech/milvus-backup/core.(*BackupContext).RestoreBackup\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:335\ngithub.com/zilliztech/milvus-backup/cmd.glob..func7\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/restore.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/runner/go/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918\ngithub.com/zilliztech/milvus-backup/cmd.Execute\n\t/home/runner/work/milvus-backup/milvus-backup/cmd/root.go:35\nmain.main\n\t/home/runner/work/milvus-backup/milvus-backup/main.go:24\nruntime.main\n\t/opt/hostedtoolcache/go/1.18.10/x64/src/runtime/proc.go:250"]
workerpool: execute job no binlog to import, input=[paths:"files/my_backup/binlogs/insert_log/452629157848001989/452629157848001990/452629157848002002/" paths:"" ]: invalid parameter
duration:0 s

Expected Behavior

all collections to be present in my DBs

Steps To Reproduce

1. backing up (works) with `./milvus-backup create -n my_backup`
2. restoring doesn't work, command: `./milvus-backup restore -n my_backup -s _recover`

I also tried to manually copy the data from the backup a-bucket to the milvus a-bucket before running the restore, but same result.

Environment

- milvus 2.4.9 via docker compose
- milvus-backup 0.4.23 running in a bitnami/pymilvus:2.4.6-debian-12-r2 container
- version of Milvus minio = Version of mivlus-backup minio = minio/minio:RELEASE.2023-03-20T20-16-18Z
- names: I want to backup and restore "milvus-minio-temp" to and from "milvus-backup-storage"

Anything else?

No response

lindner-tj commented 2 hours ago

Update: Restoring using another milvus-instance and cross-site backup works flawlessly