Closed rom1504 closed 1 year ago
maybe something like clip_benchmark eval --dataset=mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=mscoco_generative --model=coca_ViT-L-14 --output=result.json --batch_size=256 --pretrained=model.pt
Yes the following works fine for retrieval:
clip_benchmark eval --dataset=wds/mscoco_captions --dataset_root="https://huggingface.co/datasets/clip-benchmark/wds_{dataset_cleaned}/tree/main" --task=zeroshot_retrieval --model=ViT-B-32 --output=result.json
for generative something similar with task mscoco_generative
does not seem to work, will fix it in a PR and add docs for both.
Done
eg https://github.com/LAION-AI/CLIP_benchmark#coco-captions-example + https://huggingface.co/datasets/clip-benchmark/wds_mscoco_captions2017/tree/main