search

cvlab-columbia / viper

Code for the paper "ViperGPT: Visual Inference via Python Execution for Reasoning"
Other
1.63k stars 117 forks source link

Dataset-specific API examples #34

Closed astanic closed 6 months ago

astanic commented 11 months ago

Hi,

First of all thanks for your awesome work, really inspiring and also kudos for the clean and elegant code!

I was wondering, would it be possible to release the API examples specific for each dataset, i.e. Listings 2-5?

I know the examples in the wild don't need any API examples per se and work out of the box, but I'd assume that for the benchmarks and the final numbers these in-context examples play a significant role. I'm mainly curious to understand how much "hand-holding" the model needs, i.e. how detailed/extensive these examples have to be (e.g. in comparison to the ones already given in Listings 2-5)? Also, how much do these dataset-specific examples need to "cover" the distribution of possible questions for a particular dataset?

Thanks!

surisdi commented 6 months ago

Hi, apologies for the delay. Probably not useful anymore, but we updated the code with the benchmark-specific examples.