search

facebookresearch / unibench

Python Library to evaluate VLM models' robustness across diverse benchmarks
Other
169 stars 11 forks source link

Benchmark label incorrect #11

Open Nano1337 opened 3 hours ago

Nano1337 commented 3 hours ago

In the file unibench/benchmarks_zoo/benchmarks.py, it seems like dspr_y_position is defined as transfer while it's marked as vtab in the associated README in unibench/benchmarks_zoo/README.md. What's the correct benchmark group should this eval be in?

Nano1337 commented 3 hours ago

The change has been indicated in #12