Hello, thank you for the great work! I'm checking your prompts, especially your recently released prompt for GQA (prompts/benchmark/gqa.yaml). I see two types of example programs: some for demonstrating API usage, and some as in-context demonstration. Could you clarify how you curate these examples? Are they automatically generated or manually curated, and do you have specific criteria when selecting these examples?
Also, I see that some of the example questions are from GQA training set, and a few are even from the test set (for example, "Is that blanket to the right of a pillow"). Is this a bug? Does it potentially leak the test set?
Hello, thank you for the great work! I'm checking your prompts, especially your recently released prompt for GQA (
prompts/benchmark/gqa.yaml). I see two types of example programs: some for demonstrating API usage, and some as in-context demonstration. Could you clarify how you curate these examples? Are they automatically generated or manually curated, and do you have specific criteria when selecting these examples?Also, I see that some of the example questions are from GQA training set, and a few are even from the test set (for example, "Is that blanket to the right of a pillow"). Is this a bug? Does it potentially leak the test set?