vibrantvarun
commented
1 month ago Solution:
Increment the total hit count as soon as it comes in the collector.
Code changes
- Change this variable name to collectedHits
- Replace totalHits[] with collectedHits[ ]here in the topDocs()
- Create a variable totalHits like below in the collector
@Getter protected int totalHits; - Increment totalHits here
- Replace the definition of this method to just
collector.getTotalhits() - Later in Normalization remove this method at all as we will already be getting right number of totalhits from the query phase.
Improvement in Latency
(number of shards)x(Iteration over number of unique score docs in Query Phase + Normalization Phase)
martin-gaievski
chishui
brandon-carag
What is the bug?
The total count is wrong when size is given in the search request. For example When I hit the search request with Hybrid query with url below
I get
But When I hit the search request by adding size =1 I get different hit count
the response is
How can one reproduce the bug?
What is the expected behavior?
Number of total hits should be consistent.
Do you have any additional context?
The reason why this bug is coming because we calculate totalhits per shard by identifying the number of unique docId's in the result here. The query result per shard is based on numHits here. The numHits is nothing but the size sent in the search request. So for every shard it will get only 1 doc in the query result. But, collector will get the complete result irrespective of what size is given in the search request. size parameter aka numhits is always used to cut the result into size what user wants.
Let's try understand with an example
I have created my-nlp-index which has 3 shards. I have given 2 subqueries and size=1 in the search request.
Shard 1 Consider, the collector will collect the matching results(The count of the results is 40) but the when topDocs will be determined from the matching results it size would be 1 per subquery. Remaining 39 documents won't be added in the topDocs. Later, when we calculate total hits we will just calculate the unique docId's on the shard from both the subqueries. Consider that unique docId count to be 2.
Shard 2 If the same count(40) of results we got on shard 2 as well and the unique docId count is 2.
Therefore, with the current workflow the search response would have 4 as total hits. But, the actual total hit is 80.