search

orrzohar / LOVM

[NeurIPS 2023] Official Pytorch code for LOVM: Language-Only Vision Model Selection
https://orrzohar.github.io/projects/lovm/
20 stars 0 forks source link

Missing class name. #3

Closed YCaigogogo closed 11 months ago

YCaigogogo commented 11 months ago

Hello author, I found that the class name of the file clevr_closest_object_distance.txt is missing a line, I would like to ask what is the missing label?

YCaigogogo commented 11 months ago

And whether it is convenient to provide your Captions Dataset and Synonyms Dataset generated using GPT.

orrzohar commented 11 months ago

Hi @YCaigogogo,

  1. Thank you for pointing that out! I see it is an issue that goes all the way back to the CLIP benchmark's webdataset (https://huggingface.co/datasets/clip-benchmark/wds_vtab-clevr_closest_object_distance/blob/main/classnames.txt)
  2. Sure, send me an email (orrzohar@stanford.edu) I can send you these files! You can also generate your own from the provided scripts in LOVM/modelGPT.

Best, Orr

YCaigogogo commented 11 months ago

Okay, thank you for the reply. I've also noticed that the datasets in the benchmark sometimes include flowers and sometimes include food-101, which doesn't seem to be consistent. For instance, in datasets.txt there are only flowers, but in the classnames folder, there is only Food101. Are there any bugs?

orrzohar commented 11 months ago

Hi @YCaigogogo, This is a bug. I fixed it in this PR: https://github.com/orrzohar/LOVM/pull/4

In the future, is it better to open separate issues per topic so future users can find common concerns when they come up. Hope this helps! Orr

YCaigogogo commented 11 months ago

OK, Thank you very much!