erikbern / ann-benchmarks

Benchmarks of approximate nearest neighbor libraries in Python
http://ann-benchmarks.com
MIT License
4.73k stars 715 forks source link

Fix Hamming distance test to reflect normalized values #520

Closed cococo2000 closed 1 month ago

cococo2000 commented 1 month ago

Description:

This PR updates the test_hamming function in our pytest suite to correctly reflect the normalized Hamming distance. The previous test was expecting a raw Hamming distance of 2, but since our metric function calculates the normalized Hamming distance, the expected value should be 0.5.

    # ann-benchmarks/ann_benchmarks/distance.py: line 29
    "hamming": Metric(
        distance=lambda a, b: np.mean(a.astype(np.bool_) ^ b.astype(np.bool_)),
        distance_valid=lambda a: True
    ),

Changes:

Updated the expected values in test_hamming from 2 to 0.5 to align with the normalized Hamming distance calculation. Reasoning: The metrics["hamming"].distance function calculates the normalized Hamming distance by taking the mean of the boolean XOR results. Thus, for arrays p and q given in the tests:

p = [1, 1, 0, 0] q = [1, 0, 0, 1] The raw Hamming distance is 2 (two differing positions), and the normalized Hamming distance is 2/4 = 0.5.

maumueller commented 1 month ago

Thanks!

GreateFang commented 1 month ago

Hello coco @cococo2000 ! I am a database developer. Recently, I need to use ann_benchmark to test the performance of mainstream vector databases. I noticed that you recently updated the Milvus part of ann_benchmark. Have you verified that this part can produce results? Due to development needs, I need to conduct offline testing in a CentOS environment. Can the Milvus after this submission achieve this? I have been trying for a long time to run the Milvus testing part before your commits, but it didn't work.

cococo2000 commented 1 month ago

Hello coco @cococo2000 ! I am a database developer. Recently, I need to use ann_benchmark to test the performance of mainstream vector databases. I noticed that you recently updated the Milvus part of ann_benchmark. Have you verified that this part can produce results? Due to development needs, I need to conduct offline testing in a CentOS environment. Can the Milvus after this submission achieve this? I have been trying for a long time to run the Milvus testing part before your commits, but it didn't work.

I have tested the Milvus part of ann_benchmark on Ubuntu, and it has successfully produced results. Additionally, it has passed the GitHub Actions tests. image However, I have not verified it on CentOS specifically. You might need to make minor adjustments based on your specific environment.