search

mlfoundations / open_clip

An open source implementation of CLIP.
Other
9.29k stars 923 forks source link

Question: How/where to evaluate GeoDE and Dollar Street? #839

Closed gregor-ge closed 4 months ago

gregor-ge commented 4 months ago

Hello,

thank you for the great work of benchmarking all those CLIP models here: https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_classification_results.csv

I am interested in the evaluation on GeoDE and Dollar Street datasets and wanted to compare with your setup there. Afaik, you use CLIP-Benchmark (https://github.com/LAION-AI/CLIP_benchmark) for evaluating the models but I cannot find those two datasets mentioned anywhere in the repository.

Can you point me to where to find the setup you used for evaluation here? Thank you very much!

gregor-ge commented 4 months ago

I found the wds versions of the two datasets with labels for zero-shot by @djghosh13 : https://huggingface.co/datasets/djghosh/wds_dollar_street_test and https://huggingface.co/datasets/djghosh/wds_geode_test. This answers my question.