search

sandrohanea / whisper.net

Whisper.net. Speech to text made simple using Whisper Models
MIT License
534 stars 82 forks source link

is there a possibility to add CUDA or OpenCL support to whisper.net? #37

Closed GFNiko closed 1 year ago

GFNiko commented 1 year ago

I'd love to use whisper.net with a graphics card. Waiting half an hour every time I run the code gets a tad tedious after some time. Is there a possibility how we can add graphics cores and mem support?

Anyways, thanks for the port :)

drasticactions commented 1 year ago

If you build the libraries yourself, you can reference them and it should work. Build the libraries and either link them to your application or pass them in as parameters and load them dynamically, https://github.com/sandrohanea/whisper.net/blob/main/Whisper.net/WhisperFactory.cs#L28

sandrohanea commented 1 year ago

A nice option would be to have a new package Whisper.net.Runtime.Cuda where CUDA support is added and will work the same as the current Runtime. Similar for OpenCL.

drasticactions commented 1 year ago

I agree. Currently, the problem is building CUDA/OpenCL/CoreML into the pipelines. The current binaries are built on our local machines, which "works." Still, if you add those other libraries, anyone wanting to update this would need the entire dependency list for all platforms.

Ideally, this could be put into CI where all of the libraries can be generated and put into their own artifacts, then we could download them and put them into the runtime folders in this repo, or publish them outright through CI from somewhere else.

drasticactions commented 1 year ago

I am working on doing builds of the other frameworks, https://github.com/drasticactions/Whisper.net.Runtimes, with the intent of bringing it back here once I can make GitHub actions to build and publish it.

I should have CoreML figured out, although that requires an extra dylib to be loaded. That's fine if it's linked in, but for console apps without the TFM, there would need to be some extra code to load it in (You can't load the whisper dylib without the coreml one if you built it with both, bad things will happen.) Eventually when I can think of something I can make a PR for it.

sandrohanea commented 1 year ago

Hello all, Cublas support and Clblast were added for Windows in Whisper.net.Runtime.Clblast and Whisper.net.Runtime.Cublas. https://github.com/sandrohanea/whisper.net/releases/tag/1.4.4