Problem
In Disk Vector Mode, while calculating the score for a binary quantized vector, the system was incorrectly using Inner Product as the similarity function. However, for binary quantized vectors, the correct distance metric should be Hamming Distance.
Impact:
This incorrect scoring mechanism led to poor results, particularly in cases with multiple segments. During the process of reducing to the top k candidates, the best candidates (those with the most relevant results) were being filtered out. As a result, the recall was zero, meaning no relevant results were being returned, which critically affected the accuracy of search results.
Expected Behavior:
The system should use Hamming Distance for binary quantized vectors, ensuring that the correct similarity metric is applied. This would allow the best candidates to be retained during the top-k reduction process, thereby maintaining high recall and improving overall search performance in multi-segment scenarios.
Change 2:
Problem:
The system already had a setting to disable shard-level rescoring, allowing for rescoring at the segment level. However, with a recent PR, a default oversampling factor was reintroduced to be applied when the user did not explicitly provide one. Unfortunately, this default oversampling factor was being applied in both cases — regardless of whether shard-level rescoring was enabled or disabled. With Segment level , we don't need high oversampling factor.
Solution:
This PR introduces changes to ensure that the default oversampling factor is only applied when shard-level rescoring is disabled. When segment-level rescoring is enabled, the system overrides the default oversampling factor, ensuring the correct sampling factor is applied based on dimesion.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Description
Change1
Problem In Disk Vector Mode, while calculating the score for a binary quantized vector, the system was incorrectly using Inner Product as the similarity function. However, for binary quantized vectors, the correct distance metric should be Hamming Distance.
Impact: This incorrect scoring mechanism led to poor results, particularly in cases with multiple segments. During the process of reducing to the top k candidates, the best candidates (those with the most relevant results) were being filtered out. As a result, the recall was zero, meaning no relevant results were being returned, which critically affected the accuracy of search results.
Expected Behavior: The system should use Hamming Distance for binary quantized vectors, ensuring that the correct similarity metric is applied. This would allow the best candidates to be retained during the top-k reduction process, thereby maintaining high recall and improving overall search performance in multi-segment scenarios.
Change 2:
Problem: The system already had a setting to disable shard-level rescoring, allowing for rescoring at the segment level. However, with a recent PR, a default oversampling factor was reintroduced to be applied when the user did not explicitly provide one. Unfortunately, this default oversampling factor was being applied in both cases — regardless of whether shard-level rescoring was enabled or disabled. With Segment level , we don't need high oversampling factor.
Solution: This PR introduces changes to ensure that the default oversampling factor is only applied when shard-level rescoring is disabled. When segment-level rescoring is enabled, the system overrides the default oversampling factor, ensuring the correct sampling factor is applied based on dimesion.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.