search

sandrohanea / whisper.net

Whisper.net. Speech to text made simple using Whisper Models
MIT License
512 stars 78 forks source link

Add CoreML support #59

Closed drasticactions closed 1 year ago

drasticactions commented 1 year ago

This PR adds new scripts and a nuget nuspec for Whisper CoreML support. No additional APIs are required for this to work, just the runtime files and the mlmodelc files for your whisper models. You can read about that support here: https://github.com/ggerganov/whisper.cpp#core-ml-support

The biggest difference from a user perspective is that if you wish to use CoreML support, you need to build the mlmodelc files yourself, and you have to load your whisper model from a file rather than a buffer.

The mlmodelc files are, at the moment, not shared by the whisper.cpp project (they did at one point, but the repo hosting them is either removed or hidden). There are some hosted elsewhere on hugging face, but IMO, unless we built and hosted them ourselves, I feel bad linking to random builds by others.

The file usage is a limitation of whisper.cpp, which searches for the mlmodelc file based on the model path. Using the buffer would mean no file path, so it would always fail to load. Until that changes, there's not much we can do here.

sandrohanea commented 1 year ago

Hello again @drasticactions, This looks great!

Thanks for the continuous contributions and support across many platforms. Will test it asap and will also probably add some examples in https://github.com/sandrohanea/whisper.net/tree/main/examples

sandrohanea commented 1 year ago

Tested it on my mac (pretty old one) without the new ML chips but it was working :) Indeed, I will probably host the models on hugging face, and create some downloader (same as the regular models). Thanks again for the contribution!