search

harsha-simhadri / big-ann-benchmarks

Framework for evaluating ANNS algorithms on billion scale datasets.
https://big-ann-benchmarks.com
MIT License
352 stars 117 forks source link

Expanding to a full YFCC-100M filtered dataset #236

Open landrumb opened 1 year ago

landrumb commented 1 year ago

We are looking to test our submission on a 100M scale filtered dataset, and would be happy to integrate it into datasets.py if the embeddings and metadata were added to the domain where the dataset currently downloads from. We would prepare them ourselves, but the corresponding file for dataset preparation refers to an external script for generating the metadata, and we do not have the full set of CLIP descriptors.

@mdouze could you make the full 100M vector dataset available where the 10M subset is hosted at https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/yfcc100M/?

mdouze commented 11 months ago

I could probably do it but it's still smaller than 100M (like 90M or the like) because of missing images, videos, etc. LMK if this is of interest to you.

landrumb commented 11 months ago

Definitely still interested.

If we're adding other sub-100M filter datasets, do you think we should try to standardize on a round number like 50M or just subset as needed for comparison?

mdouze commented 11 months ago

Stay tuned, I have it on my TODO list...