search

harsha-simhadri / big-ann-benchmarks

Framework for evaluating ANNS algorithms on billion scale datasets.
https://big-ann-benchmarks.com
MIT License
313 stars 103 forks source link

Update leaderboard results (Streaming) #288

Closed arron2003 closed 2 months ago

arron2003 commented 3 months ago

Per conversation in #286, separate streaming leaderboard update from the original ScaNN PR.

magdalendobson commented 2 months ago

I just finished re-running every single streaming algorithm with the corrected recall computation, and my results agree with yours:

cufe,"diskann(('R32_L50', {'Ls': 70, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.1743693351745605,4886392.0,0.0,0.0,streaming,0.584525296875 cufe,"diskann(('R32_L70', {'Ls': 70, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.3363685607910156,4885996.0,0.0,0.0,streaming,0.6280998906249999 cufe,"diskann(('R50_L50', {'Ls': 70, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.5138330459594727,4885584.0,0.0,0.0,streaming,0.6481847187499999 diskann,"diskann(('R32_L50', {'Ls': 70, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.0090949535369873,4888048.0,0.0,0.0,streaming,0.664196796875 diskann,"diskann(('R32_L70', {'Ls': 70, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.0452346801757812,4887084.0,0.0,0.0,streaming,0.7042683281250001 diskann,"diskann(('R50_L50', {'Ls': 70, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.058459997177124,4885896.0,0.0,0.0,streaming,0.72150353125 hwtl_sdu_anns_stream,"HWTL_SDU_ANNS_stream(('R65_L70', {'Ls': 100, 'T': 16}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,1.0412211418151855,4887040.0,0.0,0.0,streaming,0.769358078125 pinecone,"pinecone(('R32_L100', {'Ls': 300, 'k_1': 30, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.44964075088500977,2010136.0,0.0,0.0,streaming,0.8792984374999999 pinecone,"pinecone(('R32_L100', {'Ls': 400, 'k_1': 30, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.5977978706359863,2010196.0,0.0,0.0,streaming,0.8977126875 pinecone,"pinecone(('R32_L100', {'Ls': 500, 'k_1': 30, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.4760725498199463,2010188.0,0.0,0.0,streaming,0.9104433437499999 pinecone,"pinecone(('R32_L100', {'Ls': 520, 'k_1': 30, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.4755425453186035,2012272.0,0.0,0.0,streaming,0.9120122343750001 puck,"Puck('C200_F200_FN8_Flat_filter_topk1200', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.0420689582824707,162020.0,0.0,0.0,streaming,0.08013546875000001 puck,"Puck('C200_F200_FN8_Flat_filter_topk1800', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.04240989685058594,162484.0,0.0,0.0,streaming,0.080622875 puck,"Puck('C200_F200_FN8_Flat_filter_topk1500', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.0414884090423584,162152.0,0.0,0.0,streaming,0.08365264062500001 puck,"Puck('C200_F200_FN8_Flat_filter_topk1900', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.042151689529418945,162104.0,0.0,0.0,streaming,0.08497718750000001 puck,"Puck('C200_F200_FN8_Flat_filter_topk2300', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.04342079162597656,162112.0,0.0,0.0,streaming,0.08687278125 puck,"Puck('C200_F200_FN8_Flat_filter_topk2100', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.042818546295166016,162408.0,0.0,0.0,streaming,0.0902274375 puck,"Puck('C200_F200_FN8_Flat_filter_topk2200', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.046465158462524414,162116.0,0.0,0.0,streaming,0.09184689062499998 puck,"Puck('C200_F200_FN8_Flat_filter_topk2000', {'radius_rate': 1.0, 'search_coarse_count': 200})",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.0447077751159668,162460.0,0.0,0.0,streaming,0.09214789062499999 pyanns,"pyanns(('R32_L100', {'Ls': 300, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.5063211917877197,1994584.0,0.0,0.0,streaming,0.8696431718750002 pyanns,"pyanns(('R32_L100', {'Ls': 400, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.5399785041809082,1992652.0,0.0,0.0,streaming,0.8865686875 scann,"ScaNN,tree=700/5000,AH2,reorder=317",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.0023641586303710938,1024.0,0.0,0.0,streaming,0.9924002968749999 zilliz,"zilliz(('R32_L110', {'Ls': 400, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.4923396110534668,1996968.0,0.0,0.0,streaming,0.905939296875 zilliz,"zilliz(('R32_L110', {'Ls': 450, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.490079402923584,1997400.0,0.0,0.0,streaming,0.91261615625 zilliz,"zilliz(('R32_L110', {'Ls': 500, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.48936939239501953,1997708.0,0.0,0.0,streaming,0.917919640625 zilliz,"zilliz(('R32_L110', {'Ls': 550, 'T': 8}))",msturing-30M-clustered(final_runbook.yaml),10,0.0,0.4806337356567383,1996332.0,0.0,0.0,streaming,0.9222139687499998

If you are fine with this I will ask Harsha to publish the website updates.

arron2003 commented 2 months ago

Thanks for confirming the results :) It will be great if Harsha can publish the website update.

arron2003 commented 2 months ago

Thanks a lot! Let me know how it goes with the OOD track.

magdalendobson commented 1 month ago

@nk2014yj @ZiaddAhmedd @KhylonWong @veaaaab @ingberam @hhy3 please note the updated leaderboard. With PR #280 we realized that the streaming recall was being computed incorrectly due to a caching bug. For the NeurIPS23 competition entrants, we will publish both the re-calculated leaderboard and the original leaderboard. We apologize for the error.