Do I actually need NVIDIA CUDA 12 rather than 11.8?

[DanMakingWithAI]

I just tried the instructions for implementing GPU Support e.g. installing NVIDIA CUDA Toolkit 11.8 and NVIDIA cuDNN 8.7.0 for CUDA 11.x, specifically, as per the readme (which involves some manual file moving around and PATH environment updating on Windows as per Nvidia's instructions at https://docs.nvidia.com/deeplearning/cudnn/installation/windows.html - restarted since then to ensure PATH updated properly everywhere and confirmed it can be found in the proper place when in my venv via which cublas64_11.dll (I use Git bash for my terminal in VSCode, but where via cmd works, too) and then the pytorch uninstall/reinstall pip commands specifying cuda 11.8

Then I tried to run a super simple test script:

from RealtimeSTT import AudioToTextRecorder

def process_text(text):
  print(text, end=" ", flush=True)

if __name__ == '__main__':
  with AudioToTextRecorder(
    spinner=False,
    model="tiny.en",
    language="en",
    # enable_realtime_transcription=True,
    realtime_model_type="tiny.en"
  ) as recorder:
    print("Say something...")
    while True:
      recorder.text(process_text)

But got this error:

Exception: Library cublas64_12.dll is not found or cannot be loaded

The stack trace stops in RealtimeSTT so I manually hunted around in the .venv files in vscode for a reference to cuda 12. The only reference matching a regex of cublas(.*)12 in ./.venv/* is the METADATA file for PyTorch i.e. .venv\Lib\site-packages\torch-2.2.2+cu118.dist-info\METADATA . Specifically:

Metadata-Version: 2.1
Name: torch
Version: 2.2.2+cu118
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
Home-page: https://pytorch.org/
Download-URL: https://github.com/pytorch/pytorch/tags
Author: PyTorch Team
Author-email: packages@pytorch.org
License: BSD-3
Keywords: pytorch,machine learning
Classifier: ...
...
Requires-Dist: nvidia-cuda-nvrtc-cu12 ==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-runtime-cu12 ==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cuda-cupti-cu12 ==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cudnn-cu12 ==8.9.2.26 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cublas-cu12 ==12.1.3.1 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cufft-cu12 ==11.0.2.54 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-curand-cu12 ==10.3.2.106 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cusolver-cu12 ==11.4.5.107 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-cusparse-cu12 ==12.1.0.106 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-nccl-cu12 ==2.19.3 ; platform_system == "Linux" and platform_machine == "x86_64"
Requires-Dist: nvidia-nvtx-cu12 ==12.1.105 ; platform_system == "Linux" and platform_machine == "x86_64"
...

I'm fairly inexperienced poking around to resolve Python's specific dependency hell 😅 so I'm not sure if this represents the cause of the issue or not.

I tried the uninstall, then pip cache purge, and then the re-install in case a cahed wheel was the issue, but still have the problem.

[KoljaB]

Also ran into that once. You can just search for the cublas64_11.dll and create a copy in the same folder and name it cublas64_12.dll. Seems torch fucked up their references with 2.2.2. Thanks for pointing out the lib causing this.

[DanMakingWithAI]

Brill! Yeah, I'd read about PyTorch dependency hell 🤣

Yep, that worked. Wow, that's lighting fast now 😄

faster whisper is happy with cuda 12 since v 1.0.0, so might be worth seeing if CUDA 12 will work and/or if it's time for a dependency uplift? 🤷‍♂️

And thanks for this great repository 😊

[KoljaB]

RealtimeSTT will probably work with CUDA 12, so CUDA 11 is more a recommendation, because I personally can't test anything on 12. Lots of other libraries some of my projects depend on still need CUDA 11, so I can't upgrade.

[KoljaB]

There's are also new beam_size parameters, try model "distil-small.en" with beam_size_realtime 1 - that's incredibly fast.

[ProunceDev]

Also ran into that once. You can just search for the cublas64_11.dll and create a copy in the same folder and name it cublas64_12.dll. Seems torch fucked up their references with 2.2.2. Thanks for pointing out the lib causing this.

You saved me bro, ur the goat