Switching between CPU and GPU at runtime

[AshD]

I am writing a Windows WPF desktop x64 app.

How do I switch between Whisper.net.Runtime and Whisper.net.Runtime.Clblast, if the end user has a GPU or not.

Is there a way that Whisper.net can dynamically switch and choose the best runtime for performance after checking if it's a Nvidia, AMD or no GPU present?

Thanks, Ash

[sandrohanea]

Hello @AshD, Currently this thing is not supported.

In order to create such a thing, the loader + some new package which includes all runtimes needs to be created.

[AshD]

Thanks Sandro. I can test this when it is ready.

The LlamaSharp team did something similar by trying to load different Llama.cpp dlls starting with Cuda12, Cuda11 to CPU variants. https://github.com/SciSharp/LLamaSharp/commit/10678a83d622e5b7cb5355e42486c24f74cdad4a

[lilhoser]

+1

[dfengpo]

+1