search

harsha-simhadri / big-ann-benchmarks

Framework for evaluating ANNS algorithms on billion scale datasets.
https://big-ann-benchmarks.com
MIT License
313 stars 103 forks source link

new leaderboard announcement #277

Closed ingberam closed 5 months ago

ingberam commented 5 months ago

announcement of the first ongoing leaderboard result, with the top submissions and link to the leaderboard. The link to the original leaderboard remains as agreed.

@harsha-simhadri please review

hhy3 commented 5 months ago

Thank you for providing such a great competition! But there are a few points that I don't understand. The rules mention that updating results is on February 1st. My understanding is that all submissions must be made before February 1st, but some participants are still updating their code on February 1st #275. Is this reasonable? Additionally, the final results, just like before, should be re-run rather than using the previous results cause some updates are updated on cloned repositories instead of pr on bigann repo. I hope to receive a response from the organizers. @harsha-simhadri @ingberam. Thanks!

ingberam commented 4 months ago

@hhy3 Hi, thank you for the comment, and congrats on the competitive submissions. Re #275, it was sent on time. We did not define Jan 31 as the last date, but Feb 1st. There were no other PRs around this date anyway. Nevertheless, maybe we should clarify this for the next iteration, I will propose March 1st AOE as the cutoff date for submitting a PR, and we will make a public announcement. As for re-running with updates to "cloned repositories" - I am not sure what you mean, I evaluated your submissions and you approved the results. There were no additional PRs.