Using CUDA

[ItsMeJacky]

How to run faster-whisper using CUDA ?

[rufinus]

cudnn libs + --device cuda

on my LXC setup i followed this article https://sluijsjes.nl/2024/05/18/coral-and-nvidia-passthrough-for-proxmox-lxc-to-install-frigate-video-surveillance-server/

you just have to include the correct libs in the container, or map damn from the host to the container.

i ended up building the cointainer myself with this dockerfile:

FROM rhasspy/wyoming-whisper:2.1.0
RUN apt update && apt install -y wget software-properties-common
RUN wget https://developer.download.nvidia.com/compute/cudnn/8.9.7.29/local_installers/12.2/cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-debian11-8.9.7.29/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN wget https://developer.download.nvidia.com/compute/cuda/repos/debian11/x86_64/cuda-keyring_1.1-1_all.deb
RUN dpkg -i cuda-keyring_1.1-1_all.deb
RUN add-apt-repository contrib && apt --allow-releaseinfo-change update
RUN apt -y install libcudnn8 libcublas-12-0
RUN rm cudnn-local-repo-debian11-8.9.7.29_1.0-1_amd64.deb

took some time to find the right versions to make it run.

[ssamjh]

👍 For official GPU support, would be nice to speed things up :)

[ssamjh]

:+1: for this feature. I already have a GPU in my server for other applications :)

[Mithras]

Also, is it possible to make it unload model based on some --keep-alive param?

[alienatedsec]

How to run faster-whisper using CUDA ?

Simple and works out of the box - https://github.com/linuxserver/docker-faster-whisper