search

cvlab-columbia / viper

Code for the paper "ViperGPT: Visual Inference via Python Execution for Reasoning"
Other
1.63k stars 117 forks source link

MCQ Evaluation #37

Closed chuyishang closed 6 months ago

chuyishang commented 8 months ago

Hi! I was wondering how to make the MCQ evaluation work - from my understanding the queries.csv file has a "possible_answers" column which takes a list of possible answers, but for some reason my results keep giving me answers outside of the specified list. For example, here is a line of my queries.csv:

index,sample_id,possible_answers,query_type,query,answer,image_name 0,0,"['purple', 'red', 'green', 'yellow']",,What color is the flower?,purple,flower.jpeg

But the returned result is always "blue", which I don't want to be an option. I was wondering if there is a way to fix this behavior.

surisdi commented 6 months ago

Hi, we added evaluation code to the repo. The best practice is probably to implement your own Dataset class (you can see the datasets that we implemented as examples) and load everything there. The queries.csv is just a very simplified way of having a generic dataset.