search

tazz4843 / whisper-rs

Rust bindings to https://github.com/ggerganov/whisper.cpp
The Unlicense
607 stars 105 forks source link

How to integrate in actix-web? #141

Open ChaoII opened 2 months ago

ChaoII commented 2 months ago

Hi, I am trying to integrate in actix-web , but when I try to pack params and model to web::Data<....> . Could you please give me some demo to implement my idea? otherwise, how to use in multithreading avoid to block other web request?

tazz4843 commented 2 months ago

See #86 for an idea on how to use the model and state. As for running without blocking, tokio has tokio::task::spawn_blocking for this exact purpose. Unsure what actix's equivalent would be.

ChaoII commented 2 months ago

thank you very much.

ChaoII commented 2 months ago

when I use mutithreading , something with wrong: model: ggml-large-v3-q5_0.bin

GGML_ASSERT: C:\Users\AC\CLionProjects\whisper-rust\target\debug\build\whisper-rs-sys-27850de1f340e604\out\whisper.cpp\ggml-cuda.cu:6742: ptr == (void *) (g_cuda_pool_addr[device] + g_cuda_pool_used[device])
error: process didn't exit successfully: `target\debug\whisper-rust.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)
chriskyndrid commented 2 months ago

@ChaoII, I use whisper across many threads without issue using this crate, even with multiple whisper context. Can you post more code than just the output error?

chriskyndrid commented 2 months ago

@tazz4843

I just upgrade to the latest whisper-rs release which appears to use whispercpp 1.5.4. I'm running into the same error as above, which I believe is related to:

https://github.com/ggerganov/whisper.cpp/issues/1814

I'm wondering if bumping to 1.5.5 will resolve this given the last comment on the thread?

This may also be worth review:

https://github.com/ggerganov/whisper.cpp/issues/1986

Although it does seem strange to access the same context/ggml backend across multiple threads at the same time, it might be worth noting in the docs that context can't be used in this way without resulting in a crash.

Update on this:

Running against this pull request which updates to 1.5.5 solves the aforementioned issue for me.