search

guinmoon / LLMFarm

llama and other large language models on iOS and MacOS offline using GGML library.
https://llmfarm.tech
MIT License
1.24k stars 76 forks source link

Eval error often appear when using llama 2 #9

Closed steveshaoucsb closed 11 months ago

steveshaoucsb commented 1 year ago

When I ask llama 2 after the it successfully generated an answer, or when it is in the process of generating long answers, it will always shows up “Eval error” and I need to restart the app to let the llama 2 work again. image

guinmoon commented 1 year ago

try increasing the context size to 2048

steveshaoucsb commented 1 year ago

try increasing the context size to 2048

Thanks! Now that error disappeared, but the new problem is that the llama 2 doesn't finished the answer and apparently aborted at some point. Here are examples:image image

guinmoon commented 1 year ago

That's weird. The dialog is terminated without an error only if llama generates an end-of-session token. If you continue the conversation, will it continue normally or with an error?

steveshaoucsb commented 1 year ago

It just continue without showing obvious error message, other than the conversation ends early. image

guinmoon commented 12 months ago

Tell me, did the update solve the problem?

steveshaoucsb commented 12 months ago

Just finished the update. The problem apparently still exists as llama2 still ends its reply early.image

guinmoon commented 12 months ago

Can you send me the text of the request? I'll try it myself.

steveshaoucsb commented 12 months ago

Yes. Here is my prompt:

Give me a Seoul travel plan

guinmoon commented 11 months ago

I think I found what the problem was. Tell me, did the update solve the problem?

steveshaoucsb commented 11 months ago

I think I found what the problem was. Tell me, did the update solve the problem?

It fixed!