Closed mehdidc closed 1 year ago
Thanks for catching that! The line.strip()
was an attempt to remove empty lines at the end of the file, not having realized that .splitlines()
already factors in trailing newlines. That can be easily fixed to simply return the lines as is with no filtering.
Fixed.
@djghosh13 when comparing the results of benchmark with local files vs WDS equivalent, I noticed that for
wds/vtab/clevr_closest_object_distance
, reading the classnames from https://huggingface.co/datasets/clip-benchmark/wds_vtab-clevr_closest_object_distance/blob/main/classnames.txt will cause it to ignore the 4th line because there is aif line.strip()
as a condition https://github.com/LAION-AI/CLIP_benchmark/blob/main/clip_benchmark/datasets/builder.py#L724, there it was actually meant to be empty, I was using classnames from LiT paper:Otherwise, everything else was fine.