search

sandrohanea / whisper.net

Whisper.net. Speech to text made simple using Whisper Models
MIT License
534 stars 82 forks source link

How to handle real-time sound streams #25

Open attsion opened 1 year ago

attsion commented 1 year ago

thank u

chriswalz commented 1 year ago

I'm also interested to know how use this with real-time audio streams

bmduarte commented 1 year ago

I managed to record audio chunks in real time using NAudio and iterate them as they were available. It seems to slow to handle them. Is there any way to speed things up?

VRCWizard commented 1 year ago

https://github.com/AwesomeYuer/whisper.NAudio.NET is an example but it does seem too slow for real time with this repo.

https://github.com/Const-me/Whisper/tree/master/Examples/MicrophoneCS This other implementation is much faster but it doesn't use naudio loopback (doesn't detect silence... it seems to basically chuck audio depending on what you set the capture max and min duration to be)

sandrohanea commented 1 year ago

Nice options are provided here, I am thinking that the best solution will be to have something like:

Having 2 configurable values: IntervalTime = The time needed for one processing (e.g. default to 5 seconds) OverlapTime= The time which will be processed twice in order to have continuity. (Default to 1 second)

  1. Wait for the initial {IntervalTime} and process it.
  2. Wait for additional {IntervalTime} for the second interval and process the time from {IntervalTime} - ({OverlapTime} / 2) to 2 * {IntervalTime}
  3. Identify some common segment at the end of Result1 and Begginging of Result 2 and merge them.
  4. Repeat the process.

During merge we need to keep in mind that exact the end of the segment might be gibberish, the same as the begging of the new segment (as it can be a word which is cut in half in any part).

This way, the context will be maintained as if you just cut randomly and process everything it can end up in the middle of the word and that cannot be recognized.

On the other hand, if we're always processing everything from 0 to CurrentTime, that will become too slow.

In order to improve quality, we can increase the overlap time.

Ideally, there would be this capability directly in this package, and anyone would be able to use NAudio or other stream provider to call some library (e.g. some PushStream similar to Azure Cognitive Service's PushStreams: https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/7e61fcb022f5dd75cfaf579703f8c92ad83317b0/samples/csharp/sharedcontent/console/speech_recognition_samples.cs#LL352C26-L352C26 )

dfengpo commented 2 months ago

+1