search

milvus-io / milvus

A cloud-native vector database, storage for next generation AI applications
https://milvus.io
Apache License 2.0
30.4k stars 2.91k forks source link

I properly restarted milvus (releasing collection first, then issuing docker-compose down) I loads collection just fine and say the right number of entries - but when I search for something it returns zero results.[Bug]: #18382

Closed martinenkoEduard closed 2 years ago

martinenkoEduard commented 2 years ago

Is there an existing issue for this?

Environment

I also wrote it in discussions, but it seems to be a bug, so i duplicated the topic.

Current Behavior

Screenshot 2022-07-23 092007 Screenshot 2022-07-23 092206 Screenshot 2022-07-23 092216 Screenshot 2022-07-23 092734 Screenshot 2022-07-23 093956

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [time_recorder.go:78] ["search 434763921108238337(nq=1, k=10), msgID = 434780924561588281: historical search done, msgID = 434780924561588281 (0ms)"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [query_collection.go:1091] ["streaming search start"] [msgID=434780924561588281] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [streaming.go:125] ["no partition specified, search all partitions"] [collectionID=434763921108238337] [vChannel=by-dev-rootcoord-dml_19_434763921108238337v1] ["all partitions"="[434763921108238338]"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [streaming.go:163] ["get segmentIDs by vChannel"] [collectionID=434763921108238337] [vChannel=by-dev-rootcoord-dml_19_434763921108238337v1] [partitionID=434763921108238338] [segmentIDs="[]"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [query_collection.go:1098] ["streaming search"] [msgID=434780924561588281] [collectionID=434763921108238337] ["searched dmChannel"=by-dev-rootcoord-dml_19_434763921108238337v1] ["searched partitionIDs"="[434763921108238338]"] ["searched segmentIDs"="[]"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [streaming.go:125] ["no partition specified, search all partitions"] [collectionID=434763921108238337] [vChannel=by-dev-rootcoord-dml_18_434763921108238337v0] ["all partitions"="[434763921108238338]"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [streaming.go:163] ["get segmentIDs by vChannel"] [collectionID=434763921108238337] [vChannel=by-dev-rootcoord-dml_18_434763921108238337v0] [partitionID=434763921108238338] [segmentIDs="[]"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [query_collection.go:1098] ["streaming search"] [msgID=434780924561588281] [collectionID=434763921108238337] ["searched dmChannel"=by-dev-rootcoord-dml_18_434763921108238337v0] ["searched partitionIDs"="[434763921108238338]"] ["searched segmentIDs"="[]"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [time_recorder.go:78] ["search 434763921108238337(nq=1, k=10), msgID = 434780924561588281: streaming search done, msgID = 434780924561588281 (0ms)"] milvus-standalone | [2022/07/23 06:29:20.846 +00:00] [DEBUG] [query_collection.go:1128] ["QueryNode Empty SearchResultMsg"] [collectionID=434763921108238337] [msgID=434780924561588281] [vChannels="[by-dev-rootcoord-dml_19_434763921108238337v1,by-dev-rootcoord-dml_18_434763921108238337v0]"] [sealedSegmentSearched="[]"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task_scheduler.go:681] ["got a search result"] [role=proxy] [ReqID=434780924561588281] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task_scheduler.go:701] ["first receive search result of this task"] [role=proxy] [reqID=434780924561588281] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task_scheduler.go:717] ["process search result"] [role=proxy] [collection=faces] [reqID=434780924561588281] ["answer cnt"=1] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task_scheduler.go:609] ["check if result buf is ready to reduce"] [role=proxy] [MsgID=434780924561588281] [receivedVChansSet="["by-dev-rootcoord-dml_19_434763921108238337v1","by-dev-rootcoord-dml_18_434763921108238337v0"]"] [usedVChans="["by-dev-rootcoord-dml_19_434763921108238337v1","by-dev-rootcoord-dml_18_434763921108238337v0"]"] [receivedSealedSegmentIDsSet="[]"] [receivedGlobalSegmentIDsSet="[]"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task_scheduler.go:720] ["process search result, ready to reduce"] [role=proxy] [reqID=434780924561588281] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task.go:1943] ["Proxy Search PostExecute stage1"] [len(filterSearchResults)=1] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["searchTask PostExecute: Proxy Search PostExecute stage1 done (2ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["searchTask PostExecute: decodeResultStart (0ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [proxy_session_manager.go:108] ["success to send search result"] [node=24] [base="msg_type:SearchResult msgID:434780924561588281 timestamp:434780965652922374 sourceID:24 "] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task.go:1735] [decodeSearchResults] [lenOfSearchResults=1] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["search 434763921108238337(nq=1, k=10), msgID = 434780924561588281: publish empty search result done, msgID = 434780924561588281 (1ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["search 434763921108238337(nq=1, k=10), msgID = 434780924561588281: all done, msgID = 434780924561588281 (1ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["receiveQueryMsg 434780924561588281: operation done (2ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [query_collection.go:589] ["do query done in receiveQueryMsg"] [collectionID=434763921108238337] [msgID=434780924561588281] [msgType=search] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["receiveQueryMsg 434780924561588281: all done (2ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task.go:1740] [decodeSearchResults] [i=0] [len(SlicedBob)=0] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task.go:1754] [decodeSearchResults] [lenOfResults=0] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [time_recorder.go:78] ["decodeSearchResults: done (0ms)"] milvus-standalone | [2022/07/23 06:29:20.847 +00:00] [DEBUG] [task.go:1963] ["Proxy Search PostExecute stage2"] [len(validSearchResults)=0] milvus-standalone | [2022/07/23 06:29:20.848 +00:00] [DEBUG] [task.go:1966] ["Proxy Search PostExecute stage2 failed"] [filterReason="empty search result\n"] milvus-standalone | [2022/07/23 06:29:20.848 +00:00] [DEBUG] [time_recorder.go:78] ["searchTask PostExecute: done (2ms)"] milvus-standalone | [2022/07/23 06:29:20.848 +00:00] [DEBUG] [impl.go:2500] ["Search done"] [traceID=16426306db8aaf2c] [role=proxy] [msgID=434780924561588281] [db=] [collection=faces] [partitions="[]"] [dsl=] [len(PlaceholderGroup)=2060] [OutputFields="[]"] [search_params="[{"key":"anns_field","value":"embeddings"},{"key":"topk","value":"10"},{"key":"metric_type","value":"L2"},{"key":"params","value":"{}"},{"key":"round_decimal","value":"-1"}]"] [travel_timestamp=0] [guarantee_timestamp=434780964347183104] milvus-standalone | [2022/07/23 06:29:25.622 +00:00] [DEBUG] [query_coord.go:530] ["loadBalanceSegmentLoop: memory usage rate of all online QueryNode"] ["mem rate"="{"19":0.45333381727364813}"] milvus-standalone | [2022/07/23 06:29:25.622 +00:00] [WARN] [query_coord.go:532] ["loadBalanceSegmentLoop: there are too few available query nodes to balance"] [onlineNodeIDs="[19]"] [availableNodeIDs="[19]"]

milvus-standalone | [2022/07/23 06:30:39.022 +00:00] [DEBUG] [query_collection.go:631] ["get query message from unsolvedMsg"] [collectionID=434763921108238337] [msgID=434780924561588283] [reqTime_p=2022/07/23 06:30:38.821 +00:00] [serviceTime_p=2022/07/23 06:30:38.672 +00:00] [guaranteeTime_l=434780986099892226] [serviceTime_l=434780986060832769] milvus-standalone | [2022/07/23 06:30:39.022 +00:00] [DEBUG] [query_collection.go:350] ["check if query timeout"] [collectionID=434763921108238337] [msgID=434780924561588283] [TimeoutTs=0] [curTime=434780986152583168] [timeoutTsPhysical=1970/01/01 00:00:00.000 +00:00] [curTimePhysical=2022/07/23 06:30:39.022 +00:00] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:656] ["query node::doUnsolvedMsg: add to unsolvedMsg"] [collectionID=434763921108238337] [sm.BeginTs=2022/07/23 06:30:38.821 +00:00] [serviceTime=2022/07/23 06:30:38.672 +00:00] ["delta seconds"=0] [msgID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:631] ["get query message from unsolvedMsg"] [collectionID=434763921108238337] [msgID=434780924561588283] [reqTime_p=2022/07/23 06:30:38.821 +00:00] [serviceTime_p=2022/07/23 06:30:38.871 +00:00] [guaranteeTime_l=434780986099892226] [serviceTime_l=434780986112999425] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:350] ["check if query timeout"] [collectionID=434763921108238337] [msgID=434780924561588283] [TimeoutTs=0] [curTime=434780986152845312] [timeoutTsPhysical=1970/01/01 00:00:00.000 +00:00] [curTimePhysical=2022/07/23 06:30:39.023 +00:00] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:674] ["doing search in doUnsolvedMsg..."] [collectionID=434763921108238337] [msgID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:1314] ["historical retrieve start"] [msgID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:1320] ["historical retrieve"] [msgID=434780924561588283] [collectionID=434763921108238337] ["retrieve partitionIDs"="[434763921108238338]"] ["retrieve segmentIDs"="[]"] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [time_recorder.go:78] ["retrieve 434763921108238337: historical retrieve done, msgID = 434780924561588283 (0ms)"] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:1324] ["streaming retrieve start"] [msgID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:1330] ["streaming retrieve"] [msgID=434780924561588283] [collectionID=434763921108238337] ["retrieve partitionIDs"="[434763921108238338]"] ["retrieve segmentIDs"="[]"] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [time_recorder.go:78] ["retrieve 434763921108238337: streaming retrieve done, msgID = 434780924561588283 (0ms)"] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [query_collection.go:1417] ["skip duplicated query result"] [count=0] milvus-standalone | [2022/07/23 06:30:39.023 +00:00] [DEBUG] [time_recorder.go:78] ["retrieve 434763921108238337: merge result done, msgID = 434780924561588283 (0ms)"] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [task_scheduler.go:742] ["got a retrieve result"] [role=proxy] [ReqID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [task_scheduler.go:762] ["first receive retrieve result of this task"] [role=proxy] [reqID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [task_scheduler.go:778] ["process retrieve result"] [role=proxy] [collection=faces] [reqID=434780924561588283] ["answer cnt"=1] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [task_scheduler.go:609] ["check if result buf is ready to reduce"] [role=proxy] [MsgID=434780924561588283] [receivedVChansSet="["by-dev-rootcoord-dml_19_434763921108238337v1","by-dev-rootcoord-dml_18_434763921108238337v0"]"] [usedVChans="["by-dev-rootcoord-dml_19_434763921108238337v1","by-dev-rootcoord-dml_18_434763921108238337v0"]"] [receivedSealedSegmentIDsSet="[]"] [receivedGlobalSegmentIDsSet="[]"] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [task_scheduler.go:781] ["process retrieve result, ready to reduce"] [role=proxy] [reqID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [task.go:2386] ["skip duplicated query result"] [count=0] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [INFO] [task.go:2442] ["Query result is nil"] [requestID=434780924561588283] [requestType=query] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [time_recorder.go:78] ["queryTask PostExecute: done (167ms)"] milvus-standalone | [2022/07/23 06:30:39.024 +00:00] [DEBUG] [impl.go:2718] ["Query done"] [traceID=72cef0c9566e2be8] [role=proxy] [MsgID=434780924561588283] [BeginTs=434780986099892226] [EndTs=434780986099892226] [db=] [collection=faces] [partitions="[]"] milvus-standalone | [2022/07/23 06:30:39.025 +00:00] [DEBUG] [proxy_session_manager.go:126] ["success to send retrieve result"] [node=24] [base="msg_type:RetrieveResult msgID:434780924561588283 sourceID:24 "] milvus-standalone | [2022/07/23 06:30:39.025 +00:00] [DEBUG] [query_collection.go:1366] ["QueryNode publish RetrieveResultMsg"] [msgID=434780924561588283] [vChannels="[by-dev-rootcoord-dml_19_434763921108238337v1,by-dev-rootcoord-dml_18_434763921108238337v0]"] [collectionID=434763921108238337] [sealedSegmentRetrieved="[]"] milvus-standalone | [2022/07/23 06:30:39.025 +00:00] [DEBUG] [time_recorder.go:78] ["retrieve 434763921108238337: all done, msgID = 434780924561588283 (2ms)"] milvus-standalone | [2022/07/23 06:30:39.025 +00:00] [DEBUG] [query_collection.go:706] ["do query done in doUnsolvedMsg"] [collectionID=434763921108238337] [msgID=434780924561588283] milvus-standalone | [2022/07/23 06:30:39.025 +00:00] [DEBUG] [query_collection.go:711] ["doUnsolvedMsg: do query done"] ["num of query msg"=1]

When I try to create index - i get milvus-standalone | [2022/07/23 06:33:20.324 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:20.324 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:20.324 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:20.827 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:20.828 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:20.828 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:20.828 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:21.333 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:21.334 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:21.334 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:21.334 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:21.837 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:21.838 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:21.838 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:21.838 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:22.342 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:22.343 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:22.343 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:22.344 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:22.846 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:22.847 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:22.847 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:22.847 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:23.351 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:23.352 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:23.352 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:23.353 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "] milvus-standalone | [2022/07/23 06:33:23.857 +00:00] [INFO] [impl.go:3892] ["received get flush state request"] [request="segmentIDs:434763927989256193 segmentIDs:434763931240366081 "] milvus-standalone | [2022/07/23 06:33:23.858 +00:00] [INFO] [services.go:928] ["received get flush state request"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:23.858 +00:00] [INFO] [services.go:951] ["[flush state] unflushed segment ids"] [segmentIDs="[434763927989256193,434763931240366081]"] [len=2] milvus-standalone | [2022/07/23 06:33:23.859 +00:00] [INFO] [impl.go:3906] ["received get flush state response"] [response="status:<> "]

Anything else?

Also before I encountered a similar problem - but I got only partial lost. For instance ids from 1 to 4000 were lost. but some were saved.

yanliang567 commented 2 years ago

which version of milvus are you running?

martinenkoEduard commented 2 years ago

v2.0.2, using docker-compose (standalone I suppose) standalone: container_name: milvus-standalone image: milvusdb/milvus:v2.0.2 command: ["milvus", "run", "standalone"

martinenkoEduard commented 2 years ago

I haven't delete the data yet, so I can do needed tests, if it helps to identify the bug.

martinenkoEduard commented 2 years ago

which version of milvus are you running? I am running it on ubuntu inside a VirtualBox. (but this shouldn't matter..)

Btw - all actions on the collection are successful (besides search) in Attu. I can load, release, create index (using Attu). It properly show real numbers of entities inside a collection, but it just doesn't search. I mean it returns nothing.

I suppose (haven't tried yet) it also will allow to add new elements.

martinenkoEduard commented 2 years ago

Is there a way to query all the data from collection without filtering? maybe there is something wrong with fileting?

martinenkoEduard commented 2 years ago

BTW - I just have tried to add a new vector and it successfully added it. But all previous are not present. So in only shows one element, although there must be more.

Screenshot 2022-07-23 120248

yanliang567 commented 2 years ago

@martinenkoEduard Could you please refer this script to export the whole Milvus logs for investigation? BTW， did you try release and load the collection again?

martinenkoEduard commented 2 years ago

@martinenkoEduard Could you please refer this script to export the whole Milvus logs for investigation? BTW， did you try release and load the collection again?

Yep. tried to Load and release several times. Tried to restart milvus, and even reboot the virtual machine.

martinenkoEduard commented 2 years ago

milvus.log

I haven't managed to run export-milvus-log.sh (I don't know the right parameter Specify the milvus instance name) so I unloaded logs using docker-compose logs > milvus.log

martinenkoEduard commented 2 years ago

Btw .. I tried to add one element, and then release and load the collection and this element is also gone.... milvus2.log

martinenkoEduard commented 2 years ago

Tried to add new element again and release/load collection - strangely it is here...

So to this bug to appear it has to be in memory for some time? (maybe something with flushing?) milvus3.log Screenshot 2022-07-23 124423

martinenkoEduard commented 2 years ago

This is log after milvus restart. It show only two recent elements in the collection. (

milvus_after_milvus_restart.log

martinenkoEduard commented 2 years ago

Tried to restart milvus again... Even these two are gone....

Screenshot 2022-07-23 131002 milvus_after_milvus_restart_2.log

martinenkoEduard commented 2 years ago

Yep - so the problem seems to be:

I can add new items freely. If milvus stays for a while there is a chance that some data will be "lost" It happens almost in 100% cases when you release/load a collection.

BUT you have to wait a little. If you release/load a collection right after you have added it, it will show this data..

but if you wait an then release/and load data - there is no search results........

Not sure if you have to release the collection to get this bug.

martinenkoEduard commented 2 years ago

Maybe something wrong with Consistency level? I will try on different consistency levels.

martinenkoEduard commented 2 years ago

Just tried to add new vector. added 1738 using - collection.insert([[face_id],[embedding]]) checked query all 1738 in place.

after that - released and loaded the collection and got only 546 rows!!! All other are gone... milvus.log

Screenshot 2022-07-23 172436

martinenkoEduard commented 2 years ago

Also it seems to most likely happen on 1000-1500 entries.

martinenkoEduard commented 2 years ago

tried a lot of this - all to no avail... It's not virtual box. I tried to move to real ubuntu 22.04

I don't know. Maybe it is because I insert new data when the collection is loaded? But it seems a normal thing to do.

martinenkoEduard commented 2 years ago

In my tests I found a particular thing. When I inserted a lot of random data

i = 5000 while i <2_000_000: i = i +1 rng = np.random.default_rng() arr = rng.random(512) mr = collection.insert([[i],[arr]]) if (i % 10_000) == 0:

Lost entries appeared by some reason.... They even stayed there after a restart. It is really strange...

martinenkoEduard commented 2 years ago

Addition - it also happens without load/release I just inserted data to a loaded milvus, and after 300 entries entries with ID bellow 160 became unavailable.

it just on a running instance... Without release.. milvus.log

martinenkoEduard commented 2 years ago

Also there is possibility that it is Attu's fault. When making a query it breakes something...

martinenkoEduard commented 2 years ago

Nope it only happens when running milvus using docker compose When using minikube I don't get this issue.

martinenkoEduard commented 2 years ago

So it seems to be specific to docker compose

martinenkoEduard commented 2 years ago

The same issue is appearing on ttps://github.com/milvus-io/milvus/releases/download/v2.0.2/milvus_2.0.2-1_amd64.deb And docker compose but on minikube it seems to work fine.

martinenkoEduard commented 2 years ago

It seems to be fixed with When you get collection.num_entities lost entries suddenly appear again...

connections.connect("default", host="192.168.56.102", port="19530") collection = Collection("faces")

print("") print(f"count: {collection.num_entities}") print("")

yanliang567 commented 2 years ago

@congqixia could you please take a look at this issue? it sounds like a known issue in v2.0.2 /assign @congqixia /unassign

martinenkoEduard commented 2 years ago

Yep the problem is than collection.num_entities is not instantaneous

xiaofan-luan commented 2 years ago

/assign @xaxys could you help investigate on it

xaxys commented 2 years ago

Which consistency level did you use? When you release and reload collection, milvus may need to load the data again, which costs some time. So low consistency level might cause data missing. Try CONSISTENCY_STRONG for highest consistency level.

xaxys commented 2 years ago

For first test case (i.e. row_count: 3111, milvus.log, milvus2.log), there is a known bug which is already fixed by https://github.com/milvus-io/milvus/pull/16470

martinenkoEduard commented 2 years ago

Which consistency level did you use? When you release and reload collection, milvus may need to load the data again, which costs some time. So low consistency level might cause data missing. Try CONSISTENCY_STRONG for highest consistency level.

I tried all consistency levels. Even on strong consistency level this bug appears. It appears only - if you installed milvus using docker-compose or milvus_2.0.2-1_amd64.deb

To this bug to appear it is not necessary to release/load a collection. It happened to me even when collection already was loaded. But if you release/load it happens almost in 100% cases.

So I add vectors one by one (using face detector in compreface) and after 1000 or maybe more after I load/release a collection first 1000 of vectors are gone. They can't be searched in any way.

I waited long enough after load, they didn't appear..

But if I do print(f"count: {collection.num_entities}") they appear again.

xaxys commented 2 years ago

Yes, there is a known bug occured in your log, it cause some data can not be loaded properly. I would recommend you to use milvus 2.1.0, which have fixed this bug and will be released later today.

martinenkoEduard commented 2 years ago

Yes, there is a known bug occured in your log, it cause some data can not be loaded properly. I would recommend you to use milvus 2.1.0, which have fixed this bug and will be released later today.

Du you mean -milvus-2.1.0-testing-20220725? Is there only way to build it from source code?

xaxys commented 2 years ago

Yes, there is a known bug occured in your log, it cause some data can not be loaded properly. I would recommend you to use milvus 2.1.0, which have fixed this bug and will be released later today.

Du you mean -milvus-2.1.0-testing-20220725? Is there only way to build it from source code?

No, all kinds of distributions will be provided. Sorry to have kept you waiting for so long. There is still a PR blocking the official release of 2.1.0. It will be released asap. (definetely within this week). Or, you can try dev image immediately.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.