search

Macoron / whisper.unity

Running speech to text model (whisper.cpp) in Unity3d on your local machine.
MIT License
419 stars 94 forks source link

Missing Feature! #58

Closed ehabtawfikelbwab closed 1 year ago

ehabtawfikelbwab commented 1 year ago

Hello there,

I need to give the model a hint by couple of words to get better result . How i can do that ?

Also i need an small or tiny Arabic model for mobile. Can you help me please ?

Thank you!

Macoron commented 1 year ago

I need to give the model a hint by couple of words to get better result . How i can do that ?

Check Initial Prompt field in WhisperManager. You can try to put words there, separated by commas. It doesn't always work, so you might want to experiment with different models weights and prompt varaints.

Also i need an small or tiny Arabic model for mobile.

tiny.bin and small.bin are multilingual. They should support Arabic. Try to force Arabic language by changing Language field in WhisperManager to "ar".

I don't know if there are any distilled networks fine-tuned for Arabic. OpenAI only provided English fine-tunes.

ehabtawfikelbwab commented 1 year ago

What do you mean by Initial Prompt doesn't always work? do you mean it is up to model can support that or not ? and if there a model support Initial Prompt it will always work or what ?

also can you give me link of models please

Macoron commented 1 year ago

What do you mean by Initial Prompt doesn't always work? do you mean it is up to model can support that or not ?

All models support Initial Prompt, however it's not always work as intended. For instance, if you type something like "LLAMA" in your prompt, whisper still can transcribe it as "lama".

You would need to experiment to find right prompt which works with your model.

also can you give me link of models please

Check repo readme. It should have all models there.

ehabtawfikelbwab commented 1 year ago

All models support Initial Prompt, however it's not always work as intended. For instance, if you type something like "LLAMA" in your prompt, whisper still can transcribe it as "lama".<

This is weird, Is there any update can fix that ? because I'm using custom names like "Ehab, Ihab" / "Mahmoud, Mahmud" etc. so the transcribe will be like a correction and that not good because I'm using this transcribe result later to check if exist or not (bool)

Check repo readme. It should have all models there.<

Thank you very much.

Macoron commented 1 year ago

This is weird, Is there any update can fix that ?

Your examples should work fine with a bigger networks. If you need to check names, you can try to use all names variations or do commands classification, like here #52.

I don't know any other solution except retraining network for your specific domain. You can try to ask in whisper.cpp or OpenAI repository. Maybe they have some ideas.

ehabtawfikelbwab commented 1 year ago

Okay, Thank you! feel free to close this.