search

stlukey / whispercpp.py

Python bindings for whisper.cpp
MIT License
218 stars 99 forks source link

Realtime streaming example? #1

Open milsun opened 1 year ago

milsun commented 1 year ago

Hey, can you provide an example, as to how we can run stream example as provided in whisper.cpp, with your code in python. Thanks!!!

stlukey commented 1 year ago

I haven't really explored the streaming functionality. However, from a brief look achieving this doesn't seem very straight forward. You'd need to look at whisper.cpp/stream/stream.cpp and attempt to reimplement in Cython some of the functionality which interfaces with SDL for the audio capture (could be attempted in pure Python, but it would be easier to deal with the low level data structures). Alternativley you could keep most of stream.cpp and make and expose a new single extern (i.e. transcribe_stream()) a black box, which cython can pick up on and expose to Python, where everything else is handled in c++. But you'd need to figure out a nice way to return the text back to Python.

I'm not quite sure how the example works, unfortunately I can't delve particularly deep right now, but a particular issue will be having the data sent back continuously.

A simple yet inelegant method is simply launching a subprocess instead comprising a modified stream.cpp and capturing stdout in Python.

stlukey commented 1 year ago

After having to work with stream.cpp, I see the microphone data is retrieved from SDL iteratively, in parts which are then transcribed by whisper.cpp.

The current implementation will need a way to grab so many seconds of microphone audio data. So it can be transcribed, then all the microphone data needs to be grabbed since the last time, before transcribing again etc.

EricKong1985 commented 1 year ago

I would like to get update for this topic