Sometimes crashes with UTF8 error

mdrokz / rust-llama.cpp

LLama.cpp rust bindings

https://crates.io/crates/llama_cpp_rs/

MIT License

290 stars 42 forks source link

Sometimes crashes with UTF8 error #22

Open kimtore opened 8 months ago

kimtore commented 8 months ago

Some prompts fail with:

thread 'main' panicked at 'called Result::unwrap() on an Err value: Utf8Error { valid_up_to: 0, error_len: None }', /home/kimt/.cargo/registry/src/index.crates.io-6f17d22bba15001f/llama_cpp_rs-0.3.0/src/lib.rs:528:46

I'm using this model: https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGUF

Anything I can do to try to fix this problem?

markcda commented 8 months ago

Yes, see this commit in fork.

kimtore commented 8 months ago

@markcda thanks for the workaround. Are some words now going to be dropped from the model's outputif they contain illegal UTF-8 characters?

markcda commented 7 months ago

Are some words now going to be dropped from the model's outputif they contain illegal UTF-8 characters?

The above commit only made specific deployment of the bug. Roughly speaking, the response will be reset if it contains at least one incorrect character. But the application will not crash. To solve your specific problem, I advise you to read the following post.