search

jianfch / stable-ts

Transcription, forced alignment, and audio indexing with OpenAI's Whisper
MIT License
1.59k stars 176 forks source link

Basic Real Time Streaming Capability #284

Open kanjieater opened 10 months ago

kanjieater commented 10 months ago

I'm wondering if there's any functionality that stable-ts could support in terms of streaming audio and showing it in real time. There's a demo here of something to this effect:

https://github.com/davabase/whisper_real_time/blob/master/transcribe_demo.py

jianfch commented 10 months ago

Stable-ts does not have any "real time" functionality but can be used with methods similar to the one you linked.

The method transcribes the audio in few second chunks and skips the chunks below a threshold. An obvious limitation is the high latency. The transcribe function treats the audio as complete, so there is preprocessing/postprocessing overhead that can be avoided if the function was written to treat audio as a continuous stream. Another limitation is that chunks can split in the middle of a word which can cause the model to hallucinate or skip words. Determining the boundary of a word is less straightforward when the audio has other sounds that are not speech. Since one of the focuses this repo is make Whisper reliable, it will have to use another method that does not suffer from the aforementioned limitations when implementing a real time function.