Creating a medium tflite model

[arunesh]

Hi, I tried using your collab notebook here: https://colab.research.google.com/github/nyadla-sys/whisper.tflite/blob/main/models/generate_tflite_from_whisper.ipynb

to create a medium multilingual model. When I run it with the Android app along with the multlingual vocab file filters_vocab_multilingual.bin, it doesn't recognize any words. Is there anything I could be missing ? How should I debut this.. thanks for your help in advance !

[vilassn]

Is your model loaded successfully in Android? Do you see any error or failure in logcat?

Does your model works properly in python?

[AbelAditya]

Using the medium model from https://github.com/usefulsensors/openai-whisper/blob/main/models/whisper-medium.tflite worked for me. Hope it helps you too @arunesh

[arunesh]

Hi @AbelAditya and @vilassn , Thanks for your kind reply. Its very helpful.

I tried your medium.tflite file and I see the same issue. The only word it recognized is "and" regardless of the audio. Switching back to the whisper-tiny-en.tflite works perfectly. Logcat attached.

Could it be something to do with multi-lingual and the android app in this repo, namely here: https://github.com/nyadla-sys/whisper.tflite/tree/main/android_example ?

Or maybe I got the vocab wrong, I am using filters_vocab_multilingual.bin file.

I will try the python code to see if that works with the medium files, but would love to get your thoughts in the mean time.

Thanks so much !

logcat.txt

[vilassn]

Please check whether you are using correct vocab file or not.

For multilingual model, filters_vocab_multilingual.bin is used. And for english only model, filters_vocab_gen.bin is used.

Modify isMultilingual variable as per your model type in below path TranscriptionThread.java --> run() --> boolean isMultilingual

[arunesh]

Thanks for your response @vilassn

Yes I believe, thats how I have it.

[arunesh]

Not sure what happened to the screenshot I attached to my previous reply, but here is a second attempt.

[vilassn]

@arunesh try with and without multilingual by changing isMultilingual variable. If it still doesn't work, share model path.

[AbelAditya]

I went back to the project a little while ago and found that although the model is running it is not generating accurate transcription. @arunesh Sincere apologies for misinforming you. @vilassn I tried it in both both states i.e. with isMultilingual set to true and set to false, but failed to get correct transcription. The new whisper-medium model is placed in the same directory as the other tflite models.

Please advise on how to proceed further.

[vilassn]

@AbelAditya can you please share what is expected output and what is actual output? If possible, please share audio file.

[AbelAditya]

Here is the screenshot of the transcription generated.

jfk.zip You may find the audio file used for the transcription in the attached zip folder.

This is the screenshot of the transcription generated by the tiny model of the same audio file.

isMultilingual was set to false for the above executions

[vilassn]

@AbelAditya

1. Please try the notebook below. It provides two methods for generating a TensorFlow Lite (tflite) model.
2. I have tested it with the tiny model, and it works for both methods. You can try it with the medium model.

https://github.com/nyadla-sys/whisper.tflite/blob/main/notebooks/generate_tflite_from_whisper.ipynb

Test your model within this Python notebook to verify its functionality.

If it works, during the integration into your Android app, remember to set the 'isMultilingual' flag according to the model used for conversion.

You can find detailed information about which models are English-only and which are multilingual in the following link: https://github.com/openai/whisper#available-models-and-languages

[arunesh]

Hi @AbelAditya and @vilassn Thanks for your replies. I will try creating a medium one.

I had a question about the tiny.en and tiny (multilingual). For the multilingual model, how do you set the language ? Is that automatically determined ? Also curious if we can get language detection results here with some hacks.

Thanks.

[haydenkaizeta]

https://github.com/nyadla-sys/whisper.tflite/blob/main/notebooks/generate_tflite_from_whisper.ipynb

No longer exists. Any advice on creating new whisper tflite models?

[nyadla-sys]

https://colab.research.google.com/github/nyadla-sys/whisper.tflite/blob/main/models/generate_tflite_from_whisper.ipynb

[SchweitzerGAO]

https://colab.research.google.com/github/nyadla-sys/whisper.tflite/blob/main/models/generate_tflite_from_whisper.ipynb

hi there, I am running this notebook but when it comes to inference, it fails with this error: Could you please help with this?

[scute0001]

I got the same problem like @SchweitzerGAO when run colab

https://colab.research.google.com/github/nyadla-sys/whisper.tflite/blob/main/models/generate_tflite_from_whisper.ipynb

Could you help with this or tell me something i miss?

[zhubarb]

Both of these issues are due to Tensorflow 2.15. Try:

!pip install transformers==4.33.0
!pip install datasets
!pip install tensorflow==2.14.0