-
I don't able to use llama across threads. How can I wrap it in mutex? Can we avoid using lifetimes in https://github.com/utilityai/llama-cpp-rs/blob/071598cfb85cb419c4390580054488d8dc731ff7/llama-cpp-…
-
Hi,
I've created a blank Rust project with a single dependency:
```toml
llama_cpp_rs = "0.3.0"
```
However, when I tried to compile it, the build failed with the following error:
```bash…
-
**Describe the bug**
Can get work on tabby 0.13.1 or 0.14.0 follow by the quick-start guide, it's just start process with the a embed model
/opt/tabby/bin/llama-server -m /data/models/TabbyML/Nom…
-
The command to run the `serach-api-server.wasm` .
```
wasmedge --dir .:. --env LLAMA_LOG="info" \
--nn-preload default:GGML:AUTO:Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf \
search-api-server.…
-
I use `whisper-rs` crate along with `llama-cpp-rs` and the program crash on
```rust
ctx.decode(&mut batch).expect("llama_decode() failed");
```
When not using `whisper-rs` it works but when ad…
-
### What happened?
I'm trying to use llama.cpp alongside whisper.cpp in Rust but I can't link the libraries because they link the same ggml symbols.
[llama-cpp-2](https://github.com/utilityai/ll…
-
**Describe the bug**
Trying to self host the TabbyML on K8s using Cloud Based API like openAI ones which have the GPUs capability but facing below error.
WARN llama_cpp_server::supervisor: cra…
-
Hugginface hub login successful
Used gemma2-27b LLM to testing:
cargo run --release -- -m "google/gemma-2-27b-it" -c
Finished release [optimized] target(s) in 0.03s
Running `target/re…
-
**Describe the bug**
I'm noticing below error with our Tabby deployment, looks like a memory error. Don't have any additional logs, since we've modified the logs to mask input, output information, th…
-
When I run glm-4-9b-chat-Q5_K_M.gguf on the Cuda 12 machine, the API server can be started successfully. However, when I send a question, the API server will crash.
The command I used to start the …