qinyiwei / InfoBench

MIT License
43 stars 6 forks source link

how do you generate response for each test LLM in the paper #3

Open yaoqi-zd opened 1 week ago

yaoqi-zd commented 1 week ago

hi, I find these actually exists two items: "input" and "instruction" for each sample in infobench.json file, like the following

{"id": "user_oriented_task_172", "input": "Crew Members", "category": "Indeed", "instruction": "Provide a list of the skills that may help you find the job.", "decomposed_questions": ["Is the generated text a list of skills?", "Are the skills in the generated text relevant to find the job in the given input?"], "subset": "Easy_set", "question_label": [["Format", "Content"], ["Content"]]}

do you directly concatenate the 'input' and 'instruction' as prompt to test the LLM?

songxiaoshuai commented 7 hours ago

I have the same issue; sometimes I find that directly connecting 'instruction' + \n +'input' isn't very smooth.