LAION-AI / CLIP_benchmark

CLIP-like model evaluation
MIT License
623 stars 80 forks source link

WDS classnames and empty lines #111

Closed mehdidc closed 1 year ago

mehdidc commented 1 year ago

@djghosh13 when comparing the results of benchmark with local files vs WDS equivalent, I noticed that for wds/vtab/clevr_closest_object_distance, reading the classnames from https://huggingface.co/datasets/clip-benchmark/wds_vtab-clevr_closest_object_distance/blob/main/classnames.txt will cause it to ignore the 4th line because there is a if line.strip() as a condition https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/builder.py#L724, there it was actually meant to be empty, I was using classnames from LiT paper:

lit

Otherwise, everything else was fine.

djghosh13 commented 1 year ago

Thanks for catching that! The line.strip() was an attempt to remove empty lines at the end of the file, not having realized that .splitlines() already factors in trailing newlines. That can be easily fixed to simply return the lines as is with no filtering.

mehdidc commented 1 year ago

Fixed.