milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.36k stars 2.91k forks source link

[Bug]: search timeout #17749

Closed ruixiu closed 2 years ago

ruixiu commented 2 years ago

Is there an existing issue for this?

Environment

- Milvus version: 2.0.2
- Deployment mode(standalone or cluster): standalone
- SDK version(e.g. pymilvus v2.0.0rc2): java v2.0.4
- OS(Ubuntu or CentOS): centos
- CPU/Memory: 8/32
- GPU: 
- Others: not ssd

Current Behavior

8400 2048 dimensional vectors are inserted into one set and 75645 256 dimensional vectors are inserted into another set. ID based queries on 2048 dimensional vectors using Attu have not responded. image

Expected Behavior

It is expected that 50000 2048 dimensional vectors can be queried normally

Steps To Reproduce

No response

Milvus Log

log.txt

Anything else?

No response

yanliang567 commented 2 years ago

@congqixia could you please take a look at this issue, i don't find 'search done' info in logs:

{"level":"DEBUG","time":"2022/06/24 00:10:58.418 +00:00","caller":"impl.go:2413","message":"Search received","traceID":"49ba27647d086d97","role":"proxy","db":"","collection":"truck_face","partitions":[],"dsl":"color==1 and logo==3","len(PlaceholderGroup)":8204,"OutputFields":["face_id","plate_number"],"search_params":[{"key":"anns_field","value":"vector"},{"key":"topk","value":"1000"},{"key":"metric_type","value":"L2"},{"key":"round_decimal","value":"-1"},{"key":"params","value":"{\"nprobe\":32}"}],"travel_timestamp":0,"guarantee_timestamp":1}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"id.go:140","message":"IDAllocator pickCanDoFunc","need":1,"total":199961,"remainReqCnt":0}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"impl.go:2454","message":"Search enqueued","traceID":"49ba27647d086d97","role":"proxy","msgID":434116338499977256,"timestamp":434118186339926017,"db":"","collection":"truck_face","partitions":[],"dsl":"color==1 and logo==3","len(PlaceholderGroup)":8204,"OutputFields":["face_id","plate_number"],"search_params":[{"key":"anns_field","value":"vector"},{"key":"topk","value":"1000"},{"key":"metric_type","value":"L2"},{"key":"round_decimal","value":"-1"},{"key":"params","value":"{\"nprobe\":32}"}],"travel_timestamp":0,"guarantee_timestamp":1}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"impl.go:93","message":"show collection start","role":"querycoord","collectionIDs":[],"msgID":434116338499977256}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"impl.go:121","message":"show collection end","role":"querycoord","collections":[434100699030031681,434100699305082881],"inMemoryPercentage":[100,100],"msgID":434116338499977256}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"task.go:1484","message":"QueryCoord show collections","collID":434100699030031681,"collections":[434100699030031681,434100699305082881]}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"task.go:1510","message":"translate output fields","OutputFields":["face_id","plate_number"]}
{"level":"DEBUG","time":"2022/06/24 00:10:58.419 +00:00","caller":"task.go:1557","message":"create query plan","dsl":"color==1 and logo==3","anns field":"vector","query info":"topk:1000 metric_type:\"L2\" search_params:\"{\\\"nprobe\\\":32}\" round_decimal:-1 "}
{"level":"DEBUG","time":"2022/06/24 00:10:58.420 +00:00","caller":"task.go:1599","message":"Proxy::searchTask::PreExecute","plan.OutputFieldIds":[101,102],"plan":"vector_anns:<field_id:106 predicates:<binary_expr:<op:LogicalAnd left:<unary_range_expr:<column_info:<field_id:104 data_type:Int8 > op:Equal value:<int64_val:1 > > > right:<unary_range_expr:<column_info:<field_id:105 data_type:Int8 > op:Equal value:<int64_val:3 > > > > > query_info:<topk:1000 metric_type:\"L2\" search_params:\"{\\\"nprobe\\\":32}\" round_decimal:-1 > placeholder_tag:\"$0\" > output_field_ids:101 output_field_ids:102 "}
{"level":"DEBUG","time":"2022/06/24 00:10:58.420 +00:00","caller":"time_recorder.go:78","message":"proxy execute search 434116338499977256: get used message stream (0ms)"}
{"level":"DEBUG","time":"2022/06/24 00:10:58.420 +00:00","caller":"time_recorder.go:78","message":"search: send message done (1ms)"}
{"level":"DEBUG","time":"2022/06/24 00:10:58.420 +00:00","caller":"task.go:1721","message":"proxy sent one searchMsg","collectionID":434100699030031681,"msgID":434116338499977256,"length of search msg":1,"timeoutTs":0}
{"level":"DEBUG","time":"2022/06/24 00:10:58.420 +00:00","caller":"time_recorder.go:78","message":"proxy execute search 434116338499977256: send search msg to message stream (0ms)"}
{"level":"DEBUG","time":"2022/06/24 00:10:58.420 +00:00","caller":"time_recorder.go:78","message":"proxy execute search 434116338499977256: done (0ms)"}

/assign @congqixia /unassign

ruixiu commented 2 years ago

image The memory usage is as follows. Is it normal? image

ruixiu commented 2 years ago

This is the way to create collections and indexes. init_milvus.txt

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.