search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.36k stars 2.91k forks source link

[Bug]: Search error: Search params check failed #17956

Closed JulyMood closed 2 years ago

JulyMood commented 2 years ago

Is there an existing issue for this?

Environment

- Milvus version: 2.0
- Deployment mode(standalone or cluster):standalone
- SDK version(e.g. pymilvus v2.0.0rc2):2.0.2
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: 4C8G
- GPU: 
- Others:

Current Behavior

When using the BIN_IVF_FLAT index, an error starts to be reported after a day of normal searching. The same code, the rebuilt table can be searched successfully

pymilvus console: pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=QueryNode search fail, reason [UnexpectedError] Assert "adapter->CheckSearch(conf, index_type, fieldindexing->indexing->index_mode())" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchOnSealed.cpp:91 => [SearchOnSealed]Search params check failed

Expected Behavior

return search result

Steps To Reproduce

1. embedding schema:
FieldSchema(name="embedding", dtype=params.VECTOR_TYPE, dim=800)
2. index
{
        "metric_type": "JACCARD",
        "index_type": "BIN_IVF_FLAT",
        "params": {"nlist": 1024}
}
3. param
{'metric_type': 'JACCARD', 'nprobe': 16}

Milvus Log

[2022/06/30 08:31:45.346 +00:00] [DEBUG] [impl.go:353] ["HasCollection received"] [traceID=32e61203d91d3d40] [role=proxy] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.347 +00:00] [DEBUG] [id.go:140] ["IDAllocator pickCanDoFunc"] [need=1] [total=155185] [remainReqCnt=0] [2022/06/30 08:31:45.347 +00:00] [DEBUG] [impl.go:384] ["HasCollection enqueued"] [traceID=32e61203d91d3d40] [role=proxy] [MsgID=434194538749275920] [BeginTS=434261958410895361] [EndTS=434261958410895361] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.348 +00:00] [DEBUG] [root_coord.go:1354] [HasCollection] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275920] [2022/06/30 08:31:45.348 +00:00] [DEBUG] [root_coord.go:1373] ["HasCollection success"] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275920] [2022/06/30 08:31:45.348 +00:00] [DEBUG] [impl.go:414] ["HasCollection done"] [traceID=32e61203d91d3d40] [role=proxy] [MsgID=434194538749275920] [BeginTS=434261958410895361] [EndTS=434261958410895361] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.350 +00:00] [DEBUG] [impl.go:353] ["HasCollection received"] [traceID=3d76f795a42c4c67] [role=proxy] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.351 +00:00] [DEBUG] [id.go:140] ["IDAllocator pickCanDoFunc"] [need=1] [total=155184] [remainReqCnt=0] [2022/06/30 08:31:45.351 +00:00] [DEBUG] [impl.go:384] ["HasCollection enqueued"] [traceID=3d76f795a42c4c67] [role=proxy] [MsgID=434194538749275921] [BeginTS=434261958410895362] [EndTS=434261958410895362] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.351 +00:00] [DEBUG] [root_coord.go:1354] [HasCollection] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275921] [2022/06/30 08:31:45.352 +00:00] [DEBUG] [root_coord.go:1373] ["HasCollection success"] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275921] [2022/06/30 08:31:45.352 +00:00] [DEBUG] [impl.go:414] ["HasCollection done"] [traceID=3d76f795a42c4c67] [role=proxy] [MsgID=434194538749275921] [BeginTS=434261958410895362] [EndTS=434261958410895362] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.353 +00:00] [DEBUG] [impl.go:632] ["DescribeCollection received"] [traceID=242524d231aac469] [role=proxy] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.354 +00:00] [DEBUG] [id.go:140] ["IDAllocator pickCanDoFunc"] [need=1] [total=155183] [remainReqCnt=0] [2022/06/30 08:31:45.354 +00:00] [DEBUG] [impl.go:656] ["DescribeCollection enqueued"] [traceID=242524d231aac469] [role=proxy] [MsgID=434194538749275922] [BeginTS=434261958410895363] [EndTS=434261958410895363] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.354 +00:00] [DEBUG] [root_coord.go:1394] [DescribeCollection] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275922] [2022/06/30 08:31:45.355 +00:00] [DEBUG] [root_coord.go:1412] ["DescribeCollection success"] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275922] [2022/06/30 08:31:45.355 +00:00] [DEBUG] [impl.go:689] ["DescribeCollection done"] [traceID=242524d231aac469] [role=proxy] [MsgID=434194538749275922] [BeginTS=434261958410895363] [EndTS=434261958410895363] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.358 +00:00] [DEBUG] [impl.go:353] ["HasCollection received"] [traceID=2b3ca3a5d46d0d5] [role=proxy] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.359 +00:00] [DEBUG] [id.go:140] ["IDAllocator pickCanDoFunc"] [need=1] [total=155182] [remainReqCnt=0] [2022/06/30 08:31:45.359 +00:00] [DEBUG] [impl.go:384] ["HasCollection enqueued"] [traceID=2b3ca3a5d46d0d5] [role=proxy] [MsgID=434194538749275923] [BeginTS=434261958410895365] [EndTS=434261958410895365] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.360 +00:00] [DEBUG] [root_coord.go:1354] [HasCollection] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275923] [2022/06/30 08:31:45.360 +00:00] [DEBUG] [root_coord.go:1373] ["HasCollection success"] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275923] [2022/06/30 08:31:45.360 +00:00] [DEBUG] [impl.go:414] ["HasCollection done"] [traceID=2b3ca3a5d46d0d5] [role=proxy] [MsgID=434194538749275923] [BeginTS=434261958410895365] [EndTS=434261958410895365] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.362 +00:00] [DEBUG] [impl.go:632] ["DescribeCollection received"] [traceID=17c0b6b7bd6032cf] [role=proxy] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.363 +00:00] [DEBUG] [id.go:140] ["IDAllocator pickCanDoFunc"] [need=1] [total=155181] [remainReqCnt=0] [2022/06/30 08:31:45.363 +00:00] [DEBUG] [impl.go:656] ["DescribeCollection enqueued"] [traceID=17c0b6b7bd6032cf] [role=proxy] [MsgID=434194538749275924] [BeginTS=434261958410895366] [EndTS=434261958410895366] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.363 +00:00] [DEBUG] [root_coord.go:1394] [DescribeCollection] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275924] [2022/06/30 08:31:45.363 +00:00] [DEBUG] [root_coord.go:1412] ["DescribeCollection success"] [role=rootcoord] ["collection name"=ANNS_ETMF_TEST_TENANT_TEXT] [msgID=434194538749275924] [2022/06/30 08:31:45.364 +00:00] [DEBUG] [impl.go:689] ["DescribeCollection done"] [traceID=17c0b6b7bd6032cf] [role=proxy] [MsgID=434194538749275924] [BeginTS=434261958410895366] [EndTS=434261958410895366] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [2022/06/30 08:31:45.367 +00:00] [DEBUG] [impl.go:2413] ["Search received"] [traceID=745480974c571036] [role=proxy] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [partitions="[]"] [dsl="type == 1"] [len(PlaceholderGroup)=2763] [OutputFields="[]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"topk\",\"value\":\"5\"},{\"key\":\"metric_type\",\"value\":\"JACCARD\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"}]"] [travel_timestamp=0] [guarantee_timestamp=434261957110136832] [2022/06/30 08:31:45.367 +00:00] [DEBUG] [id.go:140] ["IDAllocator pickCanDoFunc"] [need=1] [total=155180] [remainReqCnt=0] [2022/06/30 08:31:45.367 +00:00] [DEBUG] [impl.go:2454] ["Search enqueued"] [traceID=745480974c571036] [role=proxy] [msgID=434194538749275925] [timestamp=434261958424002561] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [partitions="[]"] [dsl="type == 1"] [len(PlaceholderGroup)=2763] [OutputFields="[]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"topk\",\"value\":\"5\"},{\"key\":\"metric_type\",\"value\":\"JACCARD\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"}]"] [travel_timestamp=0] [guarantee_timestamp=434261957110136832] [2022/06/30 08:31:45.368 +00:00] [DEBUG] [impl.go:93] ["show collection start"] [role=querycoord] [collectionIDs="[]"] [msgID=434194538749275925] [2022/06/30 08:31:45.369 +00:00] [DEBUG] [impl.go:121] ["show collection end"] [role=querycoord] [collections="[434211982151516161,434219392608501761,434216852773928961,434256882870517761,434261864760213505,434058632051032065,434078255612952577,434096837323390977]"] [inMemoryPercentage="[100,100,100,100,100,100,100,100]"] [msgID=434194538749275925] [2022/06/30 08:31:45.369 +00:00] [DEBUG] [task.go:1484] ["QueryCoord show collections"] [collID=434219392608501761] [collections="[434211982151516161,434219392608501761,434216852773928961,434256882870517761,434261864760213505,434058632051032065,434078255612952577,434096837323390977]"] [2022/06/30 08:31:45.369 +00:00] [DEBUG] [task.go:1510] ["translate output fields"] [OutputFields="[]"] [2022/06/30 08:31:45.369 +00:00] [DEBUG] [task.go:1557] ["create query plan"] [dsl="type == 1"] ["anns field"=embedding] ["query info"="topk:5 metric_type:\"JACCARD\" search_params:\"{}\" round_decimal:-1 "] [2022/06/30 08:31:45.369 +00:00] [DEBUG] [task.go:1599] [Proxy::searchTask::PreExecute] [plan.OutputFieldIds="[]"] [plan="vector_anns:<is_binary:true field_id:105 predicates:<unary_range_expr:<column_info: op:Equal value: > > query_info:<topk:5 metric_type:\"JACCARD\" search_params:\"{}\" round_decimal:-1 > placeholder_tag:\"$0\" > "] [2022/06/30 08:31:45.370 +00:00] [DEBUG] [time_recorder.go:78] ["proxy execute search 434194538749275925: get used message stream (0ms)"] [2022/06/30 08:31:45.370 +00:00] [DEBUG] [time_recorder.go:78] ["search: send message done (3ms)"] [2022/06/30 08:31:45.370 +00:00] [DEBUG] [task.go:1721] ["proxy sent one searchMsg"] [collectionID=434219392608501761] [msgID=434194538749275925] ["length of search msg"=1] [timeoutTs=0] [2022/06/30 08:31:45.370 +00:00] [DEBUG] [time_recorder.go:78] ["proxy execute search 434194538749275925: send search msg to message stream (0ms)"] [2022/06/30 08:31:45.370 +00:00] [DEBUG] [time_recorder.go:78] ["proxy execute search 434194538749275925: done (0ms)"] [2022/06/30 08:31:45.371 +00:00] [DEBUG] [query_collection.go:350] ["check if query timeout"] [collectionID=434219392608501761] [msgID=434194538749275925] [TimeoutTs=0] [curTime=434261958425575424] [timeoutTsPhysical=1970/01/01 00:00:00.000 +00:00] [curTimePhysical=2022/06/30 08:31:45.371 +00:00] [2022/06/30 08:31:45.371 +00:00] [DEBUG] [time_recorder.go:78] ["receiveQueryMsg 434194538749275925: get searchable time done (0ms)"] [2022/06/30 08:31:45.371 +00:00] [DEBUG] [query_collection.go:557] ["doing query in receiveQueryMsg..."] [collectionID=434219392608501761] [sm.GuaranteeTimestamp=2022/06/30 08:31:40.353 +00:00] [serviceTime=2022/06/30 08:31:45.115 +00:00] ["delta seconds"=18446744072] [msgID=434194538749275925] [msgType=search] [2022/06/30 08:31:45.371 +00:00] [DEBUG] [query_collection.go:1082] ["historical search start"] [msgID=434194538749275925] [2022/06/30 08:31:45.371 +00:00] [DEBUG] [historical.go:124] ["no partition specified, search all partitions"] [collectionID=434219392608501761] ["all partitions"="[434219392608501762]"] [2022/06/30 08:31:45.371 +00:00] [DEBUG] [segment.go:313] ["do search on segment"] [segmentID=434219392870907905] [segmentType=3] [2022/06/30 08:31:45.372 +00:00] [DEBUG] [segment.go:313] ["do search on segment"] [segmentID=434219392870907906] [segmentType=3] [2022/06/30 08:31:45.372 +00:00] [WARN] [cgo_helper.go:54] ["Search failed, C Runtime Exception: [UnexpectedError] Assert \"adapter->CheckSearch(conf, index_type, fieldindexing->indexing->index_mode())\" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchOnSealed.cpp:91\n => [SearchOnSealed]Search params check failed\n"] [2022/06/30 08:31:45.373 +00:00] [WARN] [cgo_helper.go:54] ["Search failed, C Runtime Exception: [UnexpectedError] Assert \"adapter->CheckSearch(conf, index_type, fieldindexing->indexing->index_mode())\" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchOnSealed.cpp:91\n => [SearchOnSealed]Search params check failed\n"] [2022/06/30 08:31:45.373 +00:00] [DEBUG] [time_recorder.go:78] ["receiveQueryMsg 434194538749275925: operation done (1ms)"] [2022/06/30 08:31:45.373 +00:00] [DEBUG] [task_scheduler.go:681] ["got a search result"] [role=proxy] [ReqID=434194538749275925] [2022/06/30 08:31:45.373 +00:00] [DEBUG] [task_scheduler.go:701] ["first receive search result of this task"] [role=proxy] [reqID=434194538749275925] [2022/06/30 08:31:45.373 +00:00] [DEBUG] [task_scheduler.go:717] ["process search result"] [role=proxy] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [reqID=434194538749275925] ["answer cnt"=1] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [task_scheduler.go:605] ["Proxy searchResultBuf readyToReduce"] [haveError=true] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [task_scheduler.go:720] ["process search result, ready to reduce"] [role=proxy] [reqID=434194538749275925] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [task.go:1943] ["Proxy Search PostExecute stage1"] [len(filterSearchResults)=0] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [proxy_session_manager.go:108] ["success to send search result"] [node=40] [base="msg_type:SearchResult msgID:434194538749275925 timestamp:434261958424002561 sourceID:40 "] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [query_collection.go:582] ["do query failed in receiveQueryMsg, publish failed query result"] [collectionID=434219392608501761] [msgID=434194538749275925] [msgType=search] [2022/06/30 08:31:45.374 +00:00] [WARN] [query_collection.go:384] ["[UnexpectedError] Assert \"adapter->CheckSearch(conf, index_type, fieldindexing->indexing->index_mode())\" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchOnSealed.cpp:91\n => [SearchOnSealed]Search params check failed"] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [time_recorder.go:78] ["searchTask PostExecute: Proxy Search PostExecute stage1 done (3ms)"] [2022/06/30 08:31:45.374 +00:00] [DEBUG] [time_recorder.go:78] ["searchTask PostExecute: done (3ms)"] [2022/06/30 08:31:45.374 +00:00] [ERROR] [task_scheduler.go:501] ["Failed to post-execute task: QueryNode search fail, reason [UnexpectedError] Assert \"adapter->CheckSearch(conf, index_type, fieldindexing->indexing->index_mode())\" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchOnSealed.cpp:91\n => [SearchOnSealed]Search params check failed\n: id 434194538749275925"] [traceID=745480974c571036] [stack="github.com/milvus-io/milvus/internal/proxy.(*taskScheduler).processTask\n\t/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:501"] [2022/06/30 08:31:45.374 +00:00] [WARN] [impl.go:2471] ["Search failed to WaitToFinish"] [error="QueryNode search fail, reason [UnexpectedError] Assert \"adapter->CheckSearch(conf, index_type, fieldindexing->indexing->index_mode())\" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchOnSealed.cpp:91\n => [SearchOnSealed]Search params check failed\n: id 434194538749275925"] [traceID=745480974c571036] [role=proxy] [msgID=434194538749275925] [db=] [collection=ANNS_ETMF_TEST_TENANT_TEXT] [partitions="[]"] [dsl="type == 1"] [len(PlaceholderGroup)=2763] [OutputFields="[]"] [search_params="[{\"key\":\"anns_field\",\"value\":\"embedding\"},{\"key\":\"topk\",\"value\":\"5\"},{\"key\":\"metric_type\",\"value\":\"JACCARD\"},{\"key\":\"params\",\"value\":\"{}\"},{\"key\":\"round_decimal\",\"value\":\"-1\"}]"] [travel_timestamp=0] [guarantee_timestamp=434261957110136832]

Anything else?

No response

yanliang567 commented 2 years ago

@JulyMood thank you for the issue. It seems that the search params is not correct. please try: search_params = {"metric_type": "JACCARD", "params": {"nprobe": 16}}

yanliang567 commented 2 years ago

/assign @JulyMood /unassign

zhuwenxing commented 2 years ago

In the deploy test, there are also some errors about search params for binary index, like BIN_FLAT when the param is {'metric_type': 'HAMMING', 'params': {'nprobe': 10}}

pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=1, reason=Search 1 failed, reason [UnexpectedError] Assert "search not support metric type: HAMMING" at /go/src/github.com/milvus-io/milvus/internal/core/src/query/SearchBruteForce.cpp:110

see https://github.com/milvus-io/milvus/runs/7156493125?check_suite_focus=true

It works well several days ago.

yanliang567 commented 2 years ago

@cydrain could you please take a look at this issue?

/assign @cydrain

longjiquan commented 2 years ago

/assign

longjiquan commented 2 years ago

Supported metric type: image

Similarity correlation: image

cydrain commented 2 years ago

duplicate with #16873 In 2.1.0, Milvus has been enhanced to report more detailed error info when param check fail

longjiquan commented 2 years ago

/unassign

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.