hamishivi
commented
3 months ago Hi, you will then be training the model to generate both questions and answers! Typically, people only want the model to learn how to answer questions, not generate them, so they just train on the generated answer. Prior work has mixed results on whether additionally training on the questions is useful: QLora found that it slightly hurt performance (https://arxiv.org/abs/2305.14314), while other work has suggested it might be helpful (https://arxiv.org/abs/2405.14394). We went for the more common option of just training on the answers alone, but this is something that you could play with if it's interesting to you!
In
encode_with_messages_format()offinetune.py, the codes ignore question tokens to avoid loss. My question is why ignore the question part? If I do not ignore question tokens, and update weights based on both loss from question and loss from answer? Will the fine-tuned model become worse than ignoring loss from question?