search

Caldarie / flutter_tflite_audio

Audio classification Tflite package for flutter (iOS & Android). Can support Google Teachable Machine models
MIT License
64 stars 26 forks source link

openai whisper #44

Open SomeCodingUser opened 2 years ago

SomeCodingUser commented 2 years ago

Hi,

could you use this library to run openai whisper with a tflite model? In the examples there are always labels provided, but for whisper there would not be any labels I think?

Thanks!

Caldarie commented 2 years ago

Hi,

I had a quick look and it seems openai whisper uses pytorch. For it to be compatible with this plugin:

  1. You’ll need to convert it to tflite. I believe it can be done if you follow the instructions here
  2. After converting the model, run the model through here to find input and output.
  3. As for the labels.. you don’t need to have it. As long as the input can receive audio, and the output returns an array then it should work.
  4. If input or output size does not match to this plugin, you may need to tinker the code to match your needs. Do let me know if you reach this stage

Also be aware that tflite does have a lot of limitations. For more information, read about operator compatibilities here. When converting from PyTorch to TensorFlow models, there may be some incompatibilities with your converted model on mobile devices

nyadla-sys commented 2 years ago

This is proof-of-concept project to create an Android app based on Whisper TFLite, which leverages the stock Android UI to show off its features. Whisper-TFLIte-Android-Example

Refer here for Whidper TFLite model https://github.com/usefulsensors/openai-whisper/blob/main/models/whisper.tflite

go thru README here https://github.com/usefulsensors/openai-whisper

Caldarie commented 2 years ago

@nyadla-sys Thanks for the share. :)

nyadla-sys commented 1 year ago

Feel free to download the openai/whisper-tiny tflite-based Android Whisper ASR APP from Google App Store.