leucome
commented
6 months ago Also related to the LLM trying to take user places. Is there a prompt that is known to reduce the chance that it happen. Even with llama.cpp sometime the LLM slip and try to impersonate the "User" by not using "User:"
Also I tried to find where the "User:" stop word it is in the code. but searching the word user give quite a lot of result. I though I found it in /src/features/chat/llamaCppChat.ts . But when i added "User" there alongside "User:" it did not make it stop at User.
I tried this
headers: headers, method: "POST", body: JSON.stringify({ stream: true, n_predict: 400, temperature: 0.85, cache_prompt: true, stop: [ "</s>", ${config("name")}:, "User:", "User"
slowsynapse
flukexp
Ok I used Amica with Ollama on CPU with model like mistral dolphin-mistral and Luna llama2. Everything looked fine.
Then I installed Kobold with rocBLAS. Then Amica started to respond in my place like, User bla bla bla.
I am 99% sure that User: was hard coded to stop the conversation and wait. The only explanation I can think of is that the delays for stopping the conversation is too long Is there no stopper maybe? With rocBLAS I get over 60T/s on q4 model. Then it seem to be enough to generate a lot of sentence before any stomping can happens.
I tried the demo and noticed that sometime the demo is able to say User: Yes in my place for yes no question then stop just after that. It seem to support my hypothesis that it do not stop, or stop fast enough for short sentence.
So it it plausible? If so how can we make the stopping function that kick in quicker when the AI try to respondent for the user?