zhaoxlpku
commented
3 months ago This problem is counterintuitive yet intriguing. It seems the effectiveness of these two prompts varies depending on the scenario. In situations where it's crucial to analyze the relationship between candidate answers, the v1.0 prompt might be beneficial. On the other hand, omitting candidate answers from the prompt could result in a more naturally flowing sentence, which might better align with the model's pre-training data and potentially lead to more accurate predictions.
if prompt_type == "v1.0": if answer is not None: prompt = f"Question: {question}\nCandidate answers: {candidate_answers}\nGold answer: {answer}" else: prompt = f"Question: {question}\nCandidate answers: {candidate_answers}\nGold answer:" elif prompt_type == "v2.0": if answer is not None: prompt = f"Question: {question}\nGold answer: {answer}" else: prompt = f"Question: {question}\nGold answer:"What we are doing is a multi-choice question. Why does the version that DOES NOT include candidate answer prompts have a higher prediction accuracy? I'm very confused about this, because if the candidate ABCD answer is not provided in the prompt, how does the LLM know which option should be selected? How is it done?