search

LAION-AI / CLIP_benchmark

CLIP-like model evaluation
MIT License
590 stars 75 forks source link

Webdatasets for Linear Probe evaluation #127

Open shrebox opened 4 months ago

shrebox commented 4 months ago

Hi, I noticed only the following web datasets work for the linear probe evaluation. It would be nice to have the following things fixed for the same:

  1. Support for the rest of the datasets to do linear probing evaluations. For example, the SUN397, the train shard file is missing from the HuggingFace datasets.

  2. Similar to zero-shot evaluation, an automatic way to run the linear probe evaluations would be nice to benchmark the models quickly.

Web datasets that worked for linear probe evaluation:

wds/imagenet1k
wds/fer2013
wds/voc2007
wds/cars
wds/fgvc_aircraft
wds/mnist
wds/stl10
wds/gtsrb
wds/renderedsst2
wds/vtab/caltech101
wds/vtab/cifar10
wds/vtab/cifar100
wds/vtab/clevr_count_all
wds/vtab/clevr_closest_object_distance
wds/vtab/diabetic_retinopathy
wds/vtab/dmlab
wds/vtab/dsprites_label_orientation
wds/vtab/dsprites_label_x_position
wds/vtab/dsprites_label_y_position
wds/vtab/dtd
wds/vtab/eurosat
wds/vtab/kitti_closest_vehicle_distance
wds/vtab/flowers
wds/vtab/pets
wds/vtab/pcam
wds/vtab/resisc45
wds/vtab/smallnorb_label_azimuth
wds/vtab/smallnorb_label_elevation
wds/vtab/svhn

Thanks! :)

CAOANJIA commented 3 months ago

I encountered the same problem that the training split of SUN397 is missing. Have you solved it?

shrebox commented 3 months ago

I encountered the same problem that the training split of SUN397 is missing. Have you solved it?

Unfortunately not, as there's missing train shard file (as mentioned above in the issue) for using the pipeline with web datasets :/