Closed unix-supremacist closed 10 months ago
I don't think this is a llama.cpp problem, because I can get -r llama
to stop generation immediately after the word llama in interactive mode, however the reverse prompt == stop prompt feature seems poorly documented over there, and has some interaction with being in interactive mode or not. I can't figure out if this is a CGO problem converting a []string to a char**. Local effort to do this differently had the same result, so I think the next thing is to get llama.cpp to log whatever it thought the reverse prompt was. If it propagates, or doesn't we should be able to isolate the issue to one side of the interface between go-llama.cpp and the C++ side of things.
Some printf hacking in binding.cpp gives:
Model loaded successfully.
>>> What is an animal like an alpaca?
Sending What is an animal like an alpaca?
llama_predict: negative prompt is
llama_predict: negative prompt: ''
llama_predict: antiprompt: 'llama'
llama_predict: number of tokens in negative prompt = 0
What is an animal like an alpaca?
An animal like an alpaca is a camelid, which is a family of animals that includes camels, llamas, and alpacas. Alpacas are domesticated animals that are native to the Andes mountains in South America. They are known for their soft, luxurious fleece and are often raised for their wool. Alpacas are smaller and more docile than llamas, and they are often used as pets or for display in zoos and animal sanctuaries.
from reading the doc
and reading the example
llama.SetStopWords("llama")
should stop predictions as soon as the prompt predicts this word correct? i can't get it to actually do that, it instead continues on.i did momentarily dive into the code to see if i could understand what this was doing, to see if i just misunderstood the docs, or to see if i could figure out what was wrong, but i couldn't really find anything of note.
for now it can be faked with this code inserted to the example