search

harsha-simhadri / big-ann-benchmarks

Framework for evaluating ANNS algorithms on billion scale datasets.
https://big-ann-benchmarks.com
MIT License
356 stars 118 forks source link

Change ScaNN submission to pin version and use all available cores. #310

Closed arron2003 closed 1 month ago

arron2003 commented 2 months ago

Change the scann configuration to work with benchmarking on bare-metal AMD machines (https://github.com/harsha-simhadri/big-ann-benchmarks/pull/306 from @sourcesync )

  1. Change the number of threads to be num_cpus.
  2. Pin ScaNN version.

The expected results in my run:

algorithm,parameters,dataset,count,qps,distcomps,build,indexsize,mean_ssd_ios,mean_latency,track,recall/ap
scann,"ScaNN,tree=27/40000,AH2,reorder=140",text2image-10M,10,120563.94772606545,0.0,2.384185791015625e-05,5194988.0,0,0,ood,0.8800800000000001
scann,"ScaNN,tree=34/40000,AH2,reorder=155",text2image-10M,10,104613.5888041144,0.0,2.09808349609375e-05,5194716.0,0,0,ood,0.8993499999999999
scann,"ScaNN,tree=35/40000,AH2,reorder=150",text2image-10M,10,102017.11204505162,0.0,2.09808349609375e-05,5194716.0,0,0,ood,0.90015
scann,"ScaNN,tree=35/40000,AH2,reorder=155",text2image-10M,10,102674.5089226592,0.0,2.09808349609375e-05,5194716.0,0,0,ood,0.900902
scann,"ScaNN,tree=36/40000,AH2,reorder=150",text2image-10M,10,104263.22144037379,0.0,2.09808349609375e-05,5194716.0,0,0,ood,0.901652
scann,"ScaNN,tree=37/40000,AH2,reorder=145",text2image-10M,10,106193.13800005772,0.0,2.09808349609375e-05,5194716.0,0,0,ood,0.9022270000000001
scann,"ScaNN,tree=38/40000,AH2,reorder=140",text2image-10M,10,103822.48977692406,0.0,2.09808349609375e-05,5194716.0,0,0,ood,0.902576
scann,"ScaNN,tree=42/40000,AH2,reorder=160",text2image-10M,10,91762.75642867034,0.0,2.288818359375e-05,5195112.0,0,0,ood,0.9140330000000001
sourcesync commented 2 months ago

Great! Do you guys think you can merge this today @magdalendobson @arron2003 ?

sourcesync commented 2 months ago

Great! Do you guys think you can merge this today @magdalendobson @arron2003 ?

Hey, nevermind my request to merge this PR today/soon. I ended up just merging @arron2003's remote main branch into my PR branch . Here are the updated rankings. Confirmed by @arron2003.