search

harsha-simhadri / big-ann-benchmarks

Framework for evaluating ANNS algorithms on billion scale datasets.
https://big-ann-benchmarks.com
MIT License
313 stars 103 forks source link

The algorithm does not have open source code #251

Closed DorritTang closed 7 months ago

DorritTang commented 7 months ago

https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/neurips23/streaming/hwtl_sdu_anns_stream/Dockerfile

In this algo, it only provides an installation package. Is it compliant with the requirements?

maumueller commented 7 months ago

This seems to be the diskann baseline with different parameters. I agree that it's odd to not just use the diskann Dockerfile in this case. What do you think @harsha-simhadri?

harsha-simhadri commented 7 months ago

Martin, agree. They seem to be reusing diskann python wrapper release 0.5.0rc3. The source for that version can be found here: https://github.com/microsoft/DiskANN/releases/tag/0.5.0.rc3.post1

amallia commented 7 months ago

They seem to be reusing diskann python wrapper release 0.5.0rc3

I have decompiled the code, and I can say that the DiskANN code has been modified. In fact, if you replace the wheel with the original one from the DiskANN repository, the code will not terminate in 1h

harsha-simhadri commented 7 months ago

@WPJiang Can you share the source for this entry and whether it has modified DiskANN. Thank you.

WPJiang commented 7 months ago

@WPJiang Can you share the source for this entry and whether it has modified DiskANN. Thank you.

The index construction part of DiskANN has been modified, the most obvious evidence is that ,as @amallia said, the original one can not terminate in 1h. The source code is considered to be available at some time in the future. Due to certain policies of our organization, we are currently unable to provide the source code. Sorry about that.

harsha-simhadri commented 7 months ago

@WPJiang Can you share the source for this entry and whether it has modified DiskANN. Thank you.

The index construction part of DiskANN has been modified, the most obvious evidence is that ,as @amallia said, the original one can not terminate in 1h. The source code is considered to be available at some time in the future. Due to certain policies of our organization, we are currently unable to provide the source code. Sorry about that.

@WPJiang Thanks for the clarification. I will make a note of this on the leaderboard.