search

ViperX7 / Alpaca-Turbo

Web UI to run alpaca model locally
GNU Affero General Public License v3.0
876 stars 92 forks source link

Short responses #43

Open Akiwuff opened 1 year ago

Akiwuff commented 1 year ago

I'm getting short and truncated responses. The generated answer stops in the middle of a sentence, usually after 60 - 90 words. CPU usage drops to 0, and I can see in the log that the response in the web interface is indeed identical to the output generated. I'm using the 13B model and I'm running on windows 10. So far, I have tried to change the temperature to 0.8, and the self_repeat to 128, with no apparent effect on the behaviour. I've got 32gb of RAM in this machine, and the memory isn't full when the generator is running.

Anyone got a hint for me how to fix this? TIA.

espressoelf commented 1 year ago

While I'm not able to provide an answer, https://github.com/ggerganov/llama.cpp/issues/71 has done a lot thinking about this. There seem to be some upstream improvements, too. Maybe they can be applied to @ViperX7's fork, too?