Core dump when trying to run llama based model (vicuna)

[dixxe]

FIrst, I hope this crate supports Vicuna if not, please share crate that does and I close this issue, thanks! Now back to the problem. I was following official guide and find it a bit out dated (llama.cpp doesn't have a convert.py) https://docs.llm-chain.xyz/docs/llama-tutorial/ My setup is arch linux. Also I don't have libclang package nor in repos nor in AUR, but I found clang package and it's fixed issue (with libclang dependency missing).

But I anyway followed it and have this code:

use llm_chain::executor;
use llm_chain::{parameters, prompt};
use llm_chain::options::*;
use llm_chain::options;

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let opts = options!(
        Model: ModelRef::from_path("./llama.cpp/models/Wizard-Vicuna-13B-Uncensored.Q2_K.gguf"), // Notice that we reference the model binary path
        ModelType: "llama",
        MaxContextSize: 512_usize,
        NThreads: 4_usize,
        MaxTokens: 0_usize,
        TopK: 40_i32,
        TopP: 0.95,
        TfsZ: 1.0,
        TypicalP: 1.0,
        Temperature: 0.8,
        RepeatPenalty: 1.1,
        RepeatPenaltyLastN: 64_usize,
        FrequencyPenalty: 0.0,
        PresencePenalty: 0.0,
        Mirostat: 0_i32,
        MirostatTau: 5.0,
        MirostatEta: 0.1,
        PenalizeNl: true,
        StopSequence: vec!["\n".to_string()]
    );
    let exec = executor!(llama, opts)?;
    let res = prompt!("I love Rust because")
        .run(
            &parameters!(),
            &exec,
        )
        .await?;
    println!("{}", res.to_immediate().await?);
    Ok(())
}

Running it with cargo run produces:

fatal runtime error: Rust cannot catch foreign exceptions
Aborted (core dumped)

I tested my model with llama-cli and it's working with it, producing good output. Here is journalctl of coredump:

Process 219982 (llm-chain-demo) of user 1000 dumped core.

                                                  Stack trace of thread 219982:
                                                  #0  0x000079e88b4a8e44 n/a (libc.so.6 + 0x94e44)
                                                  #1  0x000079e88b450a30 raise (libc.so.6 + 0x3ca30)
                                                  #2  0x000079e88b4384c3 abort (libc.so.6 + 0x244c3)
                                                  #3  0x000062ca2d2c071a n/a (/home/dixxe/Documents/Codes/RustThings/llm-chain-demo/target/debug/llm-chain-demo + 0x85471a)
                                                  #4  0x000062ca2d2bced8 n/a (/home/dixxe/Documents/Codes/RustThings/llm-chain-demo/target/debug/llm-chain-demo + 0x850ed8)
                                                  #5  0x000062ca2d2cc832 n/a (/home/dixxe/Documents/Codes/RustThings/llm-chain-demo/target/debug/llm-chain-demo + 0x860832)
                                                  #6  0x000062ca2d2b5ed6 n/a (/home/dixxe/Documents/Codes/RustThings/llm-chain-demo/target/debug/llm-chain-demo + 0x849ed6)
                                                  #7  0x000062ca2cbf9c3a n/a (/home/dixxe/Documents/Codes/RustThings/llm-chain-demo/target/debug/llm-chain-demo + 0x18dc3a)
                                                  #8  0x000079e88b439d4c __libc_start_main (libc.so.6 + 0x25d4c)
                                                  #9  0x000062ca2cbf3225 n/a (/home/dixxe/Documents/Codes/RustThings/llm-chain-demo/target/debug/llm-chain-demo + 0x187225)
                                                  ELF object binary architecture: AMD x86-64