Hallucinations and VAD [BLANK_AUDIO] Generations

[atx-barnes]

Tested with both small and tiny model sizes.

Using the Streaming example with VAD turned on etc. I've tried different settings and tried using a prompt to try and eliminate hallucinations and sound effects but to no avail or getting VAD to properly work I might be missing something because it treats the hallucinations of sounds like words so it struggles to turn on AD. Examples of outputs are below:

When I'm not talking and the background noise is low the following gets transcribed. Ideally, it would run inference in the background and only detect incoming audio from me talking, etc. [BLANK_AUDIO] [BLANK_AUDIO] [BLANK_AUDIO]

Most of the time with the tiny model, it loves to hallucinate sound effects from no audio or low background noises. (wind blowing), (clicking), (barking)

Are there any settings that I can try that would help eliminate hallucinations from no audio or static or get VAD correctly working?

Great project, excited for any future features or updates.

[Macoron]

Using the Streaming example with VAD turned on etc. I've tried different settings and tried using a prompt to try and eliminate hallucinations and sound effects but to no avail or getting VAD to properly work I might be missing something because it treats the hallucinations of sounds like words so it struggles to turn on AD.

Right now VAD isn't really working with streaming. It should be integrated into WhisperStream to ignore silent parts of audio which are causing hallucination. In theory it should also fix problem with repeating words. There was attempt to implement this, but it wasn't merged. See #29 for more context.

Are there any settings that I can try that would help eliminate hallucinations from no audio or static or get VAD correctly working?

You can try to:

1. Enable SingleSegment
2. Set LengthSec to something big, like 3000

This will basically make WhisperStream recurrently transcribe whole audio from begging to the end over and over again. Of course it will only make sense if you have really powerful CPU and relevantly small audio stream to work with.

[BaMarcy]

I can recommend Silero VAD model which has ONNX version and that's the state of the art and open source BTW

https://github.com/snakers4/silero-vad/tree/master/examples/cpp

[atx-barnes]

You can try to:

1. Enable SingleSegment
2. Set LengthSec to something big, like 3000

This will basically make WhisperStream recurrently transcribe whole audio from begging to the end over and over again. Of course it will only make sense if you have really powerful CPU and relevantly small audio stream to work with.

Using the microphone record scene with VAD stop enabled and those settings you provided seem to work a lot better. Another thing I noticed is that the mic has to be pretty good for VAD to work. When I used my webcam mic the VAD struggled to stop the recording but when I used a jack mic near my face it worked relatively well within ~1 sec. after I was done talking.

Thanks

[Macoron]

Another thing I noticed is that the mic has to be pretty good for VAD to work. When I used my webcam mic the VAD struggled to stop the recording but when I used a jack mic near my face it worked relatively well within ~1 sec. after I was done talking.

The VAD implementation is very basic. Original author of whisper.cpp recommend to use something more robust, like @BaMarcy suggested. But it is other extra dependency, which is out of this project reach.

Still you can try to play with Vad Thd and Vad Freq Thd parameters.

[Macoron]

@atx-barnes VAD support for streaming was recently merged (see #49). In my tests this reduced hallucinations drastically. You might want to check this out.