[Bug]: When constructing duplicate data using query expressions, the returned data may be incomplete.

[lentitude2tk]

Is there an existing issue for this?

• [X] I have searched the existing issues

Environment

- Milvus version: v2.3.13-835862d-
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): None
- SDK version(e.g. pymilvus v2.0.0rc2):java-sdk
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

1. I have constructed data and inserted five duplicate sets of data into Milvus, each ranging from 0 to 999.

   private void prepareData() {
       for (int i = 0; i < 5; ++i) {
           insertColumns(1);
       }
       createIndex();
       loadCollection();
   }
   
   private void insertColumns(int batchCount) {
       for (int batch = 0; batch < batchCount; ++batch) {
           List<List<Float>> vectors = CommonUtils.generateFixFloatVectors(VECTOR_DIM, NUM_ENTITIES);
   
           List<Long> ages = new ArrayList<>();
           List<Long> ids = new ArrayList<>();
           for (long i = 0L; i < NUM_ENTITIES; ++i) {
               ages.add((long) batch * NUM_ENTITIES + i);
               ids.add((long) batch * NUM_ENTITIES + i);
           }
   
           List<InsertParam.Field> fields = new ArrayList<>();
           fields.add(new InsertParam.Field(ID_FIELD, ids));
           fields.add(new InsertParam.Field(AGE_FIELD, ages));
           fields.add(new InsertParam.Field(VECTOR_FIELD, vectors));
   
           InsertParam insertParam = InsertParam.newBuilder()
                   .withCollectionName(COLLECTION_NAME)
                   .withFields(fields)
                   .build();
           R<MutationResult> response = milvusClient.insert(insertParam);
           CommonUtils.handleResponseStatus(response);
   
           R<FlushResponse> flush = milvusClient.flush(FlushParam.newBuilder().addCollectionName(COLLECTION_NAME).build());
           CommonUtils.handleResponseStatus(flush);
   
           GetCollectionStatisticsParam collectionStatisticsParam = GetCollectionStatisticsParam.newBuilder().withCollectionName(COLLECTION_NAME).build();
           R<GetCollectionStatisticsResponse> collectionStatistics = milvusClient.getCollectionStatistics(collectionStatisticsParam);
           CommonUtils.handleResponseStatus(collectionStatistics);
           GetCollStatResponseWrapper wrapper = new GetCollStatResponseWrapper(collectionStatistics.getData());
   
           System.out.printf("Finish insert batch%s, number of entities in Milvus: %s\n", batch, wrapper.getRowCount());
       }

2. When querying the data using a query expression with offset=0 and limit=900, I found that some data is missing in the returned results. For example, data points 351-354 are missing.

3. By querying through the interface or using queryIterator for a full query, it can be confirmed that data points 351-354 do actually exist.

Expected Behavior

First, it is expected that the returned data is de-duplicated, and there is no issue with that. However, the returned data should follow the correct sequence, and there should not be any data missing.

Steps To Reproduce

You can reproduce the issue by following steps 1-3

Milvus Log

No response

Anything else?

No response

[yanliang567]

/assign @congqixia /unassign

[xiaofan-luan]

@lentitude2tk could you try latest milvus and check if this is reproducible?

[xiaofan-luan]

/assign @MrPresent-Han

[lentitude2tk]

@lentitude2tk could you try latest milvus and check if this is reproducible?

OK, I will upgrade to the latest version 2.4 and run the test.

[lentitude2tk]

@xiaofan-luan I have upgraded to version 2.4.2-20240514-f48a7ff-72fc31b and conducted tests, which met expectations. From the results, it appears that Milvus 2.4 has optimized its query strategy compared to 2.3.

For the same dataset with offset=0, limit=900, and expr="", querying the data (0-999) repeated 5 times:

1. Version 2.3 could only return over 300 records, with some data missing.
2. Version 2.4 returned a complete set of 900 deduplicated records, covering the range 0-999.

[longjiquan]

/assign

[longjiquan]

already fixed by #32567 , in this pr, the reduce of query will try the best to get the limit results (availableCount < limit). https://github.com/milvus-io/milvus/blob/f76d16780b193cd42ef84abd0ec77b599b359d4f/internal/querynodev2/segments/result.go#L551

[lentitude2tk]

already fixed by #32567 , in this pr, the reduce of query will try the best to get the limit results (availableCount < limit).

https://github.com/milvus-io/milvus/blob/f76d16780b193cd42ef84abd0ec77b599b359d4f/internal/querynodev2/segments/result.go#L551

Can this handling strategy code be synchronized to version 2.3?

[longjiquan]

OK, I'll do it. @lentitude2tk