[Question] Faster-Whisper Home Assistant GPU or Tensor (Coral) Suport?

[werddomain]

I host the Faster-Whisper Docker container on my server. I did not see if it can use other hardware to speed things up?

Thanks!

[Shulyaka]

Faster-whisper supports GPU execution: https://github.com/guillaumekln/faster-whisper/#gpu-support

Coral TPU is not supported since it does not work well the recurring neural networks: https://github.com/guillaumekln/faster-whisper/issues/203

The last link however suggests TFLite version of https://github.com/usefulsensors/openai-whisper which could work, but no promises.

[Shulyaka]

Update: I doubt that a model of this size would fit into the edge TPU anyway. Your best bet is GPU. Probably even NVIDIA Jetson, but those are very expensive.

[werddomain]

Faster-whisper supports GPU execution: https://github.com/guillaumekln/faster-whisper/#gpu-support

Coral TPU is not supported since it does not work well the recurring neural networks: guillaumekln/faster-whisper#203

The last link however suggests TFLite version of https://github.com/usefulsensors/openai-whisper which could work, but no promises.

thanks for the fast response. I understand i need to create a docker image to support gpu?

[Shulyaka]

Don't know about docker. This line just suggests to set the device variable to cuda: https://github.com/rhasspy/rhasspy3/blob/11e8d3016d323a2ab1756dc68b4ba8a9f75f22a6/rhasspy3/configuration.yaml#L492

[werddomain]

do you know where i can find the makefile or the build file for the docker image rhasspy/wyoming-whisper ?

[Shulyaka]

Nope, sorry

[edurenye]

@Shulyaka you can find it here: https://github.com/rhasspy/wyoming-addons

And I added a PR to support GPU, and for the moment works for whisper but not for piper: https://github.com/rhasspy/wyoming-addons/pull/4

[jmorille]

The ticket https://github.com/SYSTRAN/faster-whisper/issues/203 say that not be possible ti run on TPU

But the project whisper-jax has done a migration with TPU Coral

• https://github.com/sanchit-gandhi/whisper-jax

If the huggingface test is correct

• https://huggingface.co/spaces/sanchit-gandhi/whisper-jax
• The performance was pretty amazing on running on a TPU

A blog article made some benchmark also:

• https://nikolas.blog/making-openai-whisper-faster/

Any implementation on TPU Coral could be an great specially for cheat hardware ?