Potentially Missing Questions

Yushi-Hu / tifa

TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering

https://tifa-benchmark.github.io/

Apache License 2.0

129 stars 7 forks source link

Potentially Missing Questions #3

Closed candacelax closed 12 months ago

candacelax commented 12 months ago

In the question_gen.py script, it looks like there are 12 in-context examples. In the paper, it says there are 15 examples. Any chance there are 2 missing?

Yushi-Hu commented 12 months ago

Good catch! In this codebase I subtract it to 12 because GPT-3.5 turbo's context window cannot support 15 examples. The original examples are here: https://drive.google.com/file/d/1i5fpXkUFgB7w3ip2TtmS1xORi828ow6-/view?usp=sharing Feel free to use them

candacelax commented 12 months ago

Ah that makes sense, thanks!