search

zilliztech / milvus-migration

15 stars 3 forks source link

The migrate tool shows "load success" when it is failed #11

Closed binbinlv closed 1 year ago

binbinlv commented 1 year ago

Is there an existing issue for this?

Environment

- Milvus source: 1.0.0-cpu-d030521-1ea92e
- Milvus target: 2.2.2
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    rocksmq

Current Behavior

The migrate tool shows "load success" when it is failed

[2023/02/08 15:30:31.824 +08:00] [INFO] [loader/milvus2x_loader.go:182] ["[Loader] begin to start bulkInsert"] [col=test01] [files="[/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy,/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/data.npy]"] [taskId=439289755800176026]
[2023/02/08 15:30:41.921 +08:00] [INFO] [dbclient/milvus2x.go:162] ["[Loader] Check bulkInsert"] [status="{\"ID\":439289755800176026,\"State\":1,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"collection\":\"test01\",\"failed_reason\":\"failed to get file size of '/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy', error:The specified key does not exist.\",\"files\":\"/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy,/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/data.npy\",\"partition\":\"_default\"},\"CollectionID\":439289755800176024,\"SegmentIDs\":null,\"CreateTs\":1675841431}"] [taskId=439289755800176026]
[2023/02/08 15:30:41.923 +08:00] [INFO] [starter/starter.go:55] ["[Loader] Load Success!"] [Cost=10.484945015]
[2023/02/08 15:30:41.923 +08:00] [INFO] [cmd/load.go:37] ["[Loader] load success! Will go to clean temp files, if clean step fail, just do clean!"]

Expected Behavior

Do not show "load success" when it is actually failed （Throw Error and exception）

Steps To Reproduce

1. setup  migration.yaml
2. ./milvus-migration dump 
3. ./milvus-migration load --> showing the "load success" message

Log

(towhee-test-env-3.8-0.9.0) binbin@binbindeMacBook-Pro milvus-migration % ./milvus-migration load
jodId is  loadQp7tNbyzSzdgwOZbllx8E
[2023/02/08 15:30:31.219 +08:00] [INFO] [dbclient/milvus2x.go:26] ["begin to new milvus2x client"] [endPoint=10.100.31.107:19530] [userSSL=false]
[2023/02/08 15:30:31.221 +08:00] [INFO] [dbclient/milvus2x.go:38] ["[Milvus2x] find username not empty, will use NewDefaultGrpcClientWithURI() to new client"]
[2023/02/08 15:30:31.342 +08:00] [INFO] [dbclient/milvus2x.go:46] ["[Milvus2x] begin to test connect"] [endpoint=10.100.31.107:19530] [username=root]
[2023/02/08 15:30:31.438 +08:00] [INFO] [meta/meta_helper.go:52] ["[MetaHelper] begin to get meta, "] [metaMode=sqlite]
[2023/02/08 15:30:31.439 +08:00] [INFO] [meta/sqlite_meta_reader.go:32] ["[Meta Reader] begin to connect sqlite"] [sqlite=/Users/binbin/milvus-1.0/milvus/db/meta.sqlite]
[2023/02/08 15:30:31.450 +08:00] [INFO] [meta/sqlite_meta_reader.go:67] ["[Meta Reader] finish read collection meta"] [allSegment=1]
[2023/02/08 15:30:31.451 +08:00] [WARN] [meta/meta_helper.go:102] ["filter cols is empty, will read all collection"]
[2023/02/08 15:30:31.451 +08:00] [INFO] [loader/milvus2x_loader.go:68] ["[Loader] load meta finish"]
[2023/02/08 15:30:31.451 +08:00] [INFO] [loader/milvus2x_loader.go:79] ["[Loader] Begin to load table."]
[2023/02/08 15:30:31.733 +08:00] [INFO] [loader/milvus2x_loader.go:169] ["[Loader] Begin to load files"] [fileKey=test011675753122650772000]
[2023/02/08 15:30:31.824 +08:00] [INFO] [loader/milvus2x_loader.go:182] ["[Loader] begin to start bulkInsert"] [col=test01] [files="[/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy,/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/data.npy]"] [taskId=439289755800176026]
[2023/02/08 15:30:41.921 +08:00] [INFO] [dbclient/milvus2x.go:162] ["[Loader] Check bulkInsert"] [status="{\"ID\":439289755800176026,\"State\":1,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"collection\":\"test01\",\"failed_reason\":\"failed to get file size of '/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy', error:The specified key does not exist.\",\"files\":\"/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy,/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/data.npy\",\"partition\":\"_default\"},\"CollectionID\":439289755800176024,\"SegmentIDs\":null,\"CreateTs\":1675841431}"] [taskId=439289755800176026]
[2023/02/08 15:30:41.922 +08:00] [ERROR] [dbclient/milvus2x.go:162] ["[Loader] Error bulkInsert"] [status="{\"ID\":439289755800176026,\"State\":1,\"RowCount\":0,\"IDList\":null,\"Infos\":{\"collection\":\"test01\",\"failed_reason\":\"failed to get file size of '/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy', error:The specified key does not exist.\",\"files\":\"/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/id.npy,/Users/binbin/milvus-migration/s3_log/migration/tables/test01/1675753122650772000/data.npy\",\"partition\":\"_default\"},\"CollectionID\":439289755800176024,\"SegmentIDs\":null,\"CreateTs\":1675841431}"] [taskId=439289755800176026] [] [stack="github.com/zilliztech/milvus-migration/core/dbclient.(*Milvus2x).WaitBulkLoadSuccess\n\t/Users/binbin/milvus-migration/core/dbclient/milvus2x.go:162\ngithub.com/zilliztech/milvus-migration/core/loader.(*Milvus2xLoader).loadDataOne\n\t/Users/binbin/milvus-migration/core/loader/milvus2x_loader.go:187\ngithub.com/zilliztech/milvus-migration/core/loader.(*Milvus2xLoader).loadDataBatch.func1\n\t/Users/binbin/milvus-migration/core/loader/milvus2x_loader.go:169\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/Users/binbin/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75"]
[2023/02/08 15:30:41.923 +08:00] [INFO] [starter/starter.go:55] ["[Loader] Load Success!"] [Cost=10.484945015]
[2023/02/08 15:30:41.923 +08:00] [INFO] [cmd/load.go:37] ["[Loader] load success! Will go to clean temp files, if clean step fail, just do clean!"]
[2023/02/08 15:30:41.923 +08:00] [INFO] [factory/factory.go:123] ["[Factory] begin to new s3 session"] [useIAM=false] [region=ap-southeast-1] [bucketName=target]
[2023/02/08 15:30:41.924 +08:00] [INFO] [factory/factory.go:83] ["[S3] Begin to list buckets"] [bucket=target]
[2023/02/08 15:30:42.042 +08:00] [INFO] [factory/factory.go:90] ["[S3] Find bucket exist, just return"] [bucket=target]
[2023/02/08 15:30:42.042 +08:00] [INFO] [cleaner/s3_cleaner.go:27] ["[S3 Cleaner] Begin to clean files"] [bucket=target] [rootPath=/Users/binbin/milvus-migration/s3_log/migration]
load error:  InvalidParameter: 1 validation error(s) found.
- missing required field, DeleteObjectsInput.Delete.Objects.

Anything else?

No response

Thomas-HuWei commented 1 year ago

Fixed!