LAION-AI / CLIP_benchmark

CLIP-like model evaluation
MIT License
535 stars 68 forks source link

WDS classnames and empty lines #111

Closed mehdidc closed 7 months ago

mehdidc commented 8 months ago

@djghosh13 when comparing the results of benchmark with local files vs WDS equivalent, I noticed that for wds/vtab/clevr_closest_object_distance, reading the classnames from https://huggingface.co/datasets/clip-benchmark/wds_vtab-clevr_closest_object_distance/blob/main/classnames.txt will cause it to ignore the 4th line because there is a if line.strip() as a condition https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/builder.py#L724, there it was actually meant to be empty, I was using classnames from LiT paper:

lit

Otherwise, everything else was fine.

djghosh13 commented 8 months ago

Thanks for catching that! The line.strip() was an attempt to remove empty lines at the end of the file, not having realized that .splitlines() already factors in trailing newlines. That can be easily fixed to simply return the lines as is with no filtering.

mehdidc commented 7 months ago

Fixed.