search

jianfch / stable-ts

Transcription, forced alignment, and audio indexing with OpenAI's Whisper
MIT License
1.47k stars 167 forks source link

Basic Real Time Streaming Capability #284

Open kanjieater opened 8 months ago

kanjieater commented 8 months ago

I'm wondering if there's any functionality that stable-ts could support in terms of streaming audio and showing it in real time. There's a demo here of something to this effect:

https://github.com/davabase/whisper_real_time/blob/master/transcribe_demo.py

jianfch commented 8 months ago

Stable-ts does not have any "real time" functionality but can be used with methods similar to the one you linked.

The method transcribes the audio in few second chunks and skips the chunks below a threshold. An obvious limitation is the high latency. The transcribe function treats the audio as complete, so there is preprocessing/postprocessing overhead that can be avoided if the function was written to treat audio as a continuous stream. Another limitation is that chunks can split in the middle of a word which can cause the model to hallucinate or skip words. Determining the boundary of a word is less straightforward when the audio has other sounds that are not speech. Since one of the focuses this repo is make Whisper reliable, it will have to use another method that does not suffer from the aforementioned limitations when implementing a real time function.