[Bug]: count* and the num of query_iter have wide gap

[lzhin]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version:2.3.5
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):pymilvus v2.3.7
- OS(Ubuntu or CentOS): Ubuntu 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

with the same expr "448526226933733731<id<448526253755733731",count and the num of query_iter have wide gap.["{'count()': 190357716}"] and the num of query_iter is 117086495 .

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[yanliang567]

@lzhin do you have any duplicated primary keys in the collection? I am asking because count(*) does not do de-dup while query_iter returns results after de-dup.

/assign @lzhin /unassign

[lzhin]

@lzhin do you have any duplicated primary keys in the collection? I am asking because count(*) does not do de-dup while query_iter returns results after de-dup.

/assign @lzhin /unassign

how to check whether has duplicated primary keys? I feel that it might not have duplicated pkeys. because I not set the pkey by myself . the pkey is auto generate by milvus

[yanliang567]

@lzhin Could you please refer this doc to export the whole Milvus logs for investigation? For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs. Also if convenient, please retry on latest milvus v2.4.4.

[lzhin]

@lzhin Could you please refer this doc to export the whole Milvus logs for investigation? For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs. Also if convenient, please retry on latest milvus v2.4.4.

I see the export tool use kubectl to export log. but I don't deploy milvus by k8s or operator, I deploy the every moudle use my company's tool , and connect them each other use domain name. so it seems can't export log use export-milvus-log.sh or use docker-compose logs. I try upgrade milvus to 2.3.13,but fail because when the minio upgrade to 2023-03-20T20-16-18Z, the minio crash. I also try the milvus-backup to backup the collection,but fail when restore the data to the new milvus. it seams that the new minio could not read the backup data. so now I try to use query_iter to export all the collection data to a file and use batch insert to new milvus.

[xiaofan-luan]

this might be a bug of query iterator and @MrPresent-Han fix that

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[xiaofan-luan]

for counter issue, recent updates is we saw duplicte data might affect count accuracy

[lzhin]

for counter issue, recent updates is we saw duplicte data might affect count accuracy

what is the mean of duplicate data, is it mean the same vector? I guess that the counter is larger than the actual vector numbers, it might cause by the crash server, some data might not seal to the minio,but had recorded in etcd.

[MrPresent-Han]

Hi, lzhin, the duplicate data mentioned above is referred to the entities with duplicated primary key, because the result returned by query iterator will exclude results with same pks but the count(*) will not, so this may cause the gap between these two cases

[lzhin]

Hi, lzhin, the duplicate data mentioned above is referred to the entities with duplicated primary key, because the result returned by query iterator will exclude results with same pks but the count(*) will not, so this may cause the gap between these two cases

hi, what's the reason that existing duplicated primary key, I think that it should not appear duplicated pk, the pk is automatic generated by milvus

[xiaofan-luan]

@bigsheeper

can we build a tool for user to analyze duplicted pk existence?