[BUG/Help] 在gsm8k上测试时, 模型总是会生成我提供的prompt中的例子, 这种情况如何避免?

[zty07]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

给的prompt Question: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? Let's think step by step Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The answer is 6.

Question: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? Let's think step by step There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The answer is 5.

Question: 真正的要解答的问题 Let\'s think step by step

Expected Behavior

No response

Steps To Reproduce

下面是模型的输出 Question: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done, there will be 21 trees. How many trees did the grove workers plant today? Let's think step by step Then there were 21 trees after some more were planted. So there must have been 21 - 15 = 6. The answer is 6.

Question: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot? Let's think step by step There are originally 3 cars. 2 more cars arrive. 3 + 2 = 5. The answer is 5.

Question: A boy has 12 oranges. He gives one-third of this number to his brother, one-fourth of the remainder to his friend and keeps the rest for himself. How many does his friend get? Let's think step by step He gives out 12 / 3 = 4 oranges. He then has 12 - 4 = 8 oranges. He gives out 8 / 4 = 2 oranges. The answer is 2.

Environment

- OS:
- Python:
- Transformers:
- PyTorch:
- CUDA Support (`python -c "import torch; print(torch.cuda.is_available())"`) :

Anything else?

No response

[duzx16]

你用的是 generate 方法还是 chat 方法，transformers 的 generate 方法返回的本来就是包含输入的

[cgq0816]

能不能提供一下GSM8K的测评代码？我这里评测都是0，谢谢