search

V-Sekai / godot-whisper

An GDExtension addon for the Godot Engine that enables realtime audio transcription, supports OpenCL for most platforms, Metal for Apple devices, and runs on a separate thread.
MIT License
70 stars 7 forks source link

Do something more for the silence mic hallucinating part #20

Closed Ughuuu closed 9 months ago

Ughuuu commented 12 months ago

It's not very bad as is, maybe we just expose the vad option threshold and play with it.

fire commented 10 months ago

@aiaimimi0920 Is the silence mic hallucination better now?

aiaimimi0920 commented 10 months ago

no, now when you not talking ,it will automatically adds "Thank you" or "you" like this issue: https://github.com/ggerganov/whisper.cpp/issues/1592

maybe I can use the method mentioned in this issue to solve it

fire commented 10 months ago

Is this fixed by the latest changes?

aiaimimi0920 commented 10 months ago

Not resolved.

  1. I just added a bloacking character map like "thanks" and "thanks you". But when using other languages, there is still a high probability of hallucinatory characters text, such as “xx字幕”，“谢谢你”

  2. Because when I check whether the audio is pure silent in the "add buffer", it will detect whether the energy is less than a certain value. If your environment is very quiet, it won't generate hallucinatory text. But if your environment is somewhat noisy, such as the sound of a fan, it will still be judged as a valid audio file entering the inference stage, and then generate generating hallucinatory text

Possible solutions:

  1. Increase the amount of blacklist: However, I am very skeptical whether it will filter out valid text
  2. Before input, make a certain judgment on whether the audio only contains environmental noise. If there is no effective human voice, filter it directly
fire commented 10 months ago

I've seen solutions like implementing an audio denoise https://github.com/snakers4/silero-vad but it seems more work to use than ggml-whisper.

aiaimimi0920 commented 10 months ago

It is a PyTorch model, and we may need a C++ implementation, or use iree.gd?

fire commented 10 months ago

I can't estimate how long it will take to combine silero-vad with Whisper together but it should be feasible inside of iree.gd.

Ughuuu commented 9 months ago

Exposed the vad option for threshold. Also made it so that if it's halucinating, only take max 4 tokens, and no more(as they could be legit characters). Anything else should fall outside of this repo(eg. processing of text with iree.gd). We can open another issue for that after iree.gd is released.