[FEAT] GPU support

[modem7]

Is this a new feature request?

• [X] I have searched the existing issues

Wanted change

Nvidia GPU support.

Reason for change

Faster-whisper is far faster with GPU.

Proposed code change

cuda runtimes as seen in https://github.com/ahmetoner/whisper-asr-webservice/blob/main/Dockerfile.gpu

[github-actions[bot]]

Thanks for opening your first issue here! Be sure to follow the relevant issue templates, or risk having this issue marked as invalid.

[TRusselo]

i came here to look into GPU support, and request if needed. found I am not alone.

[babanovac1980]

So I am not crazy :) I tried every possible combination of extra parameters on the docker container settings (--gpus all --runtime=nvidia, nvidia visible devices) but it still used the CPU and it's really slow. Got 2 Tesla P40 in my home server waiting to be used. So I second this request. Thanks for all your hard work.

[thespad]

So the short answer is that adding Nvidia runtime support will at least double the size of the image. I might look at adding a separate tag so the core image isn't burdened with it, but for example that image OP linked is an absolutely stupid 11.5Gb - at some point you might as well just run a dedicated VM for it.

[babanovac1980]

Thanks. Whisper-asr-webservice has added faster whisper as an alternate engine but his image doesn't have Wyoming protocol which is needed for Home Assistant's voice assistants. For someone looking for self hosted home automation the last thing one needs is to wait 3-4-5 seconds for whisper to process the voice on CPU so that the locally linked light switch turns on or off. Giving people an option to make their own decision if the compromise is worth it seems like the right way forward. As for a dedicated VM, it's not an option for me, and others probably as the thought of dedicating a full 24 GB tesla to whisper alone doesn't sit right with me. Seems like a big waste.

[thespad]

Please take a look at https://github.com/linuxserver/docker-faster-whisper/pull/6 once it's finished building; the image tag will be posted to the PR to allow you to test.

[babanovac1980]

Please take a look at #6 once it's finished building; the image tag will be posted to the PR to allow you to test.

At at first glance doesn't seem to work on GPUs. Nothing gets loaded into GPU ram. The new image works but still on CPU. Do I need to add any other parameters?

`docker run -d --name='faster-whisper' --net='bridge' -e TZ="Europe/Athens" -e HOST_OS="Unraid" -e HOST_HOSTNAME="ElderNet" -e HOST_CONTAINERNAME="faster-whisper" -e 'WHISPER_MODEL'='base' -e 'WHISPER_BEAM'='1' -e 'WHISPER_LANG'='en' -e 'PUID'='99' -e 'PGID'='100' -e 'UMASK'='022' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/linuxserver-ls-logo.png' -p '10300:10300/tcp' -v '/mnt/user/appdata/faster-whisper':'/config':'rw' --runtime=nvidia --gpus=all 'lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-f96940c1787b378f93093112f6bf0115e17f97fc-pr-6' c273cb14c49a87acf11d46976b752bdf8c263387e863cb9a624c188ca18f155e

The command finished successfully!`

No errors in the logs.

[thespad]

Runtime needs to be set to nvidia and you need the Nvidia container toolkit installed on the host

[babanovac1980]

It is: --runtime=nvidia I also have the Nvidia drivers plugin ( I am using Unraid - although it shouldn't matter)

I have other containers setup the same way (like Jellyfin) and they work on GPU without issues.

[thespad]

My mistake, I forgot to set the device in the service file, just fixing that now hopefully - the docs aren't great

[thespad]

New build is up, ignore the CI failure that's because the GitHub runners don't have a GPU

lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6

[babanovac1980]

docker run -d --name='faster-whisper' --net='bridge' -e TZ="Europe/Athens" -e HOST_OS="Unraid" -e HOST_HOSTNAME="ElderNet" -e HOST_CONTAINERNAME="faster-whisper" -e 'WHISPER_MODEL'='base' -e 'WHISPER_BEAM'='1' -e 'WHISPER_LANG'='en' -e 'runtime'='nvidia' -e 'PUID'='99' -e 'PGID'='100' -e 'UMASK'='022' -l net.unraid.docker.managed=dockerman -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/linuxserver-ls-logo.png' -p '10300:10300/tcp' -v '/mnt/user/appdata/faster-whisper':'/config':'rw' --runtime=nvidia --gpus=all 'lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6'

Unable to find image 'lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6' locally docker: Error response from daemon: manifest for lspipepr/faster-whisper:gpu-1.0.1-pkg-c8490d79-dev-af5c5eb79d5e4a524997696b375db083119fcf65-pr-6 not found: manifest unknown: manifest unknown. See 'docker run --help'.

The command failed.

[thespad]

Ugh, because the CI test failed it didn't push the image to the public registry.

I've kicked off a new build with the CI test disabled.

[babanovac1980]

Image can be pulled now, however I am getting:

01:23:52 PM Traceback (most recent call last): 01:23:52 PM File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main 01:23:52 PM return _run_code(code, main_globals, None, 01:23:52 PM File "/usr/lib/python3.10/runpy.py", line 86, in _run_code 01:23:52 PM exec(code, run_globals) 01:23:52 PM File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 136, in 01:23:52 PM asyncio.run(main()) 01:23:52 PM File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run 01:23:52 PM return loop.run_until_complete(main) 01:23:52 PM File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete 01:23:52 PM return future.result() 01:23:52 PM File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 112, in main 01:23:52 PM whisper_model = WhisperModel( 01:23:52 PM File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/faster_whisper/transcribe.py", line 58, in init 01:23:52 PM self.model = ctranslate2.models.Whisper( 01:23:52 PM RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

What version of Cuda Toolkit is used? As far as I know: For 2.3 you need a 190.x driver, for 3.0 you need 195.x and for 3.1 you need 256.x (actually anything up to the next multiple of five is ok, e.g. 258.x for 3.1). My drivers are reported as latest: latest: v535.54.03

[thespad]

The pip packages installed (as per the faster-whisper docs) are the latest versions of nvidia-cublas-cu11 ad nvidia-cudnn-cu11, so CUDA 11 in both cases,

I've got some hardware I can spin up later this week to be able to test locally because at the moment I'm just having to wing it based on the project docs.

[thespad]

I've run through a whole bunch of tests with both the CUDA 11 and 12 packages and multiple different drivers/runtimes and I can't get it to work. Everything results in the same error, even with the latest drivers and toolkit available.

[jaburges]

i'm happy to test also if needed - got a GPU in my home server and hopefully will see some snappy responses (size of image likely wont be an issue for those with GPUs, unless its possible to use a Coral TPU?)

My Nvidia driver on Unraid is 535.104.05

[stefanthoss]

I was able to run the a7597ab commit from the gpu-initial branch on my local machine using a Quadro P620 (Driver Version: 545.29.06, CUDA Version: 12.3). The trick was to install the NVIDIA Container Toolkit first and then exposing the GPU in Docker with

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

[markoj21]

Was also able to get the gpu branch working, Gforce 1070TI and the performance is fantastic.

image: lspipepr/faster-whisper:gpu-version-1.0.1

[thespad]

Cool, I'll get the PR reviewed and merged shortly. Thanks for the testing.

[babanovac1980]

It still doesn't work for me. Using this image: lspipepr/faster-whisper:gpu-version-1.0.1 I get:

Traceback (most recent call last): File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.10/runpy.py", line 86, in _run_code exec(code, run_globals) File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 136, in asyncio.run(main()) File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run return loop.run_until_complete(main) File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete return future.result() File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/main.py", line 112, in main whisper_model = WhisperModel( File "/lsiopy/lib/python3.10/site-packages/wyoming_faster_whisper/faster_whisper/transcribe.py", line 58, in init self.model = ctranslate2.models.Whisper( RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

[thespad]

Try passing --gpus all as part of your run. Unfortunately my spare nvidia card is an ancient GTX 780 so it's not a good test case for trying to narrow down where the issue lies.

[babanovac1980]

Try passing --gpus all as part of your run. Unfortunately my spare nvidia card is an ancient GTX 780 so it's not a good test case for trying to narrow down where the issue lies.

That did the trick. Thank you for all your hard work and time.

[ds-sebastian]

I'm trying to get GPU version working with no luck. I'm using unRAID nvidia driver plugin with the following setup:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.40.07              Driver Version: 550.40.07      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
|   0  NVIDIA GeForce RTX 2070        Off |   00000000:01:00.0 Off |                  N/A |
|  0%   58C    P2             65W /  185W |     665MiB /   8192MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

unRAID creates the container using the following run command:

docker run
  -d
  --name='faster-whisper'
  --net='pipernet'
  --privileged=true
  -e TZ="America/New_York"
  -e HOST_OS="Unraid"
  -e HOST_CONTAINERNAME="faster-whisper"
  -e 'WHISPER_MODEL'='medium-int8'
  -e 'NVIDIA_DRIVER_CAPABILITIES'='all'
  -e 'PUID'='99'
  -e 'PGID'='100'
  -e 'UMASK'='022'
  -l net.unraid.docker.managed=dockerman
  -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/linuxserver-ls-logo.png'
  -p '10300:10300/tcp'
  -v '/mnt/cache/appdata/faster-whisper':'/config':'rw' 'lscr.io/linuxserver/faster-whisper:gpu'
  --gpus all
  --runtime nvidia

However, the container logs just show the following:

  RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

I've tried multiple different tags and also the lspipepr but can't seem to get it to work 🤷 Would a new image have to be created with FROM nvidia/cuda:12.4.0-base-ubuntu22.04 so the container version matches the runtime version?

[j0nnymoe]

You don't use the gpus flag nor have you set the Nvidia runtime correctly.

[ds-sebastian]

@j0nnymoe, I'm not sure what you mean, could you explain further? I've also tried removing gpus flag and adding the =

  --runtime=nvidia

Do I need a different version of nvidia drivers? I have other containers that use my GPU (Emby, stable-diffusion, compreface, frigate) and they all seem to work as expected with those flags I typically provide.

EDIT: Ok, seems like I was adding the flag as an Post Argument not as an Extra Parameter. It works when I correct that mistake

[j0nnymoe]

You will need to add the runtime + the ENV's NVIDIA_VISIBLE_DEVICES & NVIDIA_DRIVER_CAPABILITIES.

If you continue to have issue, please either jump on our discord or forum as our GitHub issues isn't for general support. (ignoring the fact this is actually a closed issue anyway.)