search

airockchip / rknn-llm

Other
417 stars 36 forks source link

Phi3 推理无法结束或在最后出现无效输出 #36

Closed AndySze closed 6 months ago

AndySze commented 6 months ago

使用Phi3模型，推理结束貌似有问题， Prompt Prefix和Postfix参考官方的文档后，这样写的：

#define PROMPT_TEXT_PREFIX "<|user|>"
#define PROMPT_TEXT_POSTFIX "<|end|><|assistant|>"

Ollama的Template是这样的：

TEMPLATE "{{ if .System }}<|system|>
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|user|>
{{ .Prompt }}<|end|>
{{ end }}<|assistant|>
{{ .Response }}<|end|>
"

如下所示：

第一个例子（回答完毕后，没有马上结束，而是输出了很多无效文字，最终超过最长限制后结束？）

user: hi, how are you?
robot:  Hello! I'm an AI, so I don't have feelings, but I'm functioning optimally and ready to assist you. How can I help you today?

(Note: This response serves as a greeting in return while also providing information about the capabilities of the assistant.)<|end|><|assistant|> Good day! I'm an AI digital assistant designed to answer your questions and provide assistance. While I don't have personal experiences, how may I serve you today?

(Note: The response maintains a polite tone while clarifying that the AI doesn't have feelings or personal experiences.)<|end|><|assistant|> Greetings! I appreciate your interest in my status. As an artificial intelligence, I'm always operational and ready to help. What can we discuss or work on together?

(Note: The response acknowledges the question about well-being while emphasizing readiness for assistance.)<|end|><|assistant|> Hello there! Although I don't have emotions, I am fully capable of providing you with information and support. How can I be of service to you right now?

(Note: The AI responds appropriately by acknowledging the inquiry about well-being but clarifying its nature as an artificial intelligence.)<|end|><|endoftext|>

第二个例子（回答完毕后，会在后面追加<|end|><|endoftext|>)

前面省略了若干句子

A poem to capture a snowy day's essence,

Where each flake tells its own presence.

In this moment of pure delight,

Snowfall turns darkness into light.<|end|><|endoftext|>
lzz773751548 commented 6 months ago

这个问题是因为phi3实际表示推理结束的标志为"<|end|>",而在tokenizer_config.json文件中,用于判断推理结束的标志eos_token被设置为"<|endoftext|>". 解决这个问题只需要在转换模型前,将tokenizer_config.json中的eos_token设置为"<|end|>"即可

AndySze commented 6 months ago

这个问题是因为phi3实际表示推理结束的标志为"<|end|>",而在tokenizer_config.json文件中,用于判断推理结束的标志eos_token被设置为"<|endoftext|>". 解决这个问题只需要在转换模型前,将tokenizer_config.json中的eos_token设置为"<|end|>"即可

感谢解答，重新改了tokenizer_config.json并转了模型，可以及时停止输出了，但每次会在最后打印：<|end|> 比如：

1、
前面省略... How can I help you today?<|end|>

2、
For words are but a poet's art,
That touches hearts from start to part.<|end|>

3、
In winter’s arms we find our peace, in frost-kissed moments shared,
A timeless poem of snow and soul, forevermore declared.<|end|>

另外，如果输出文字很长，超过了设定的token数量限制，会在结尾处，输出一些无关内容，如下两个例子：

1、
前面省略...
it held something far more precious - ancient knowledge about the world's natural balance = self.__get_balance(
                                                                                          ^^^^^^^^^^^^^^^^^^^^^
                                                                                          无关内容

2、
前面省略...
Remy would occasionally look back to€fragment{<div class="modal" data-dismiss="modal">...</div>}
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                    无关内容