search

ggerganov / whisper.cpp

Port of OpenAI's Whisper model in C/C++
MIT License
35.81k stars 3.65k forks source link

Request for Guidance on Running whisper.cpp with cuBLAS on Windows #878

Open bbscout opened 1 year ago

bbscout commented 1 year ago

I am new to both Whisper.cpp and C++, and I would appreciate some guidance on how to run whisper.cpp with cuBLAS. I have successfully downloaded the Windows binaries (whisper-blas-bin-x64.zip) and executed main.exe using the following command:

$ ./main.exe -m ./model/ggml-large.bin -l cs -osrt audio_16khz.wav

My system has a working CUDAToolkit installed, as I can use it with PyTorch (torch.cuda.is_available() >>> true). The CUDA Toolkit is installed in the following directory: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1.

I expect the find_package(CUDAToolkit) function to locate the Toolkit and set BLAS = 0. However, I am unsure if this is happening correctly. Please forgive me if I am overlooking something obvious. Here is the console output I get when running the command:

whisper_init_from_file_no_state: loading model from './model/ggml-large.bin'
whisper_model_load: loading model
...
system_info: n_threads = 4 / 20 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 | COREML = 0 | 

main: processing 'audio_16khz.wav' (5341101 samples, 333.8 sec), 4 threads, 1 processors, lang = cs, task = transcribe, timestamps = 1 ...
matrix07012 commented 1 year ago

I believe you either need to compile it yourself or they've just added GitHub action for a cublas build so you can get the latest build here: https://github.com/ggerganov/whisper.cpp/actions/runs/4885055280

bbscout commented 1 year ago

I already downloaded generated binaries with cuBLAS support. But it is not using BLAS (BLAS = 0) while running.

savchenko commented 1 year ago

I already downloaded generated binaries with cuBLAS support. But it is not using BLAS (BLAS = 0 while running.

I have noticed that Windows blas-enabled build of v1.4.0 reports BLAS = 0 as well.

@ggerganov , is it a regression?

ggerganov commented 1 year ago

@ggerganov , is it a regression?

Looks like it - I can't test on Windows. Hopefully someone will investigate and help resolve

nalbion commented 1 year ago

I've built it locally with WHISPER_CUBLAS set and get 

System info: AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |
cddqssc commented 1 year ago

system_info: n_threads = 4 / 20 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | COREML = 0 |

Although BLAS = 1， I see in task manager that GPU(RTX 3070TI) is not working, CPU is working.

savchenko commented 1 year ago

Compiled locally, works fine: https://github.com/ggerganov/whisper.cpp/issues/89#issuecomment-1574852496

bobqianic commented 1 year ago

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

Build

Well done!!!

ShinichiYao commented 1 year ago

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

Build

  • Open CMD console
  • Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_OPENBLAS=1
  • Input cmake.exe --build whisper_build_location --config release

Well done!!!

  • All binaries can be found in whisper_build_location\bin\Release

I get this error Could NOT find BLAS (missing: BLAS_LIBRARIES) Untitled-1

bobqianic commented 1 year ago

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

Build

  • Open CMD console
  • Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_OPENBLAS=1
  • Input cmake.exe --build whisper_build_location --config release

Well done!!!

  • All binaries can be found in whisper_build_location\bin\Release

I get this error Could NOT find BLAS (missing: BLAS_LIBRARIES) Untitled-1

Oops, seems like I goofed up there. The command should be -D WHISPER_CUBLAS=1, not -D WHISPER_OPENBLAS=1. Thanks for bearing with me. Oh, and by the way, if you're looking to compile the OpenBLAS version, you'll need to download and install OpenBLAS first.

LLLYF commented 3 months ago

@cddqssc @bbscout Here is a very simple guidance to help you build whisper.cpp with CuBLAS on windows:

Preparation

Build

  • Open CMD console
  • Input cmake.exe -S whisper.cpp_source_code_location -B whisper_build_location -D WHISPER_CUBLAS=1
  • Input cmake.exe --build whisper_build_location --config release

Well done!!!

  • All binaries can be found in whisper_build_location\bin\Release

Sorry to bother you, sir, I meet some problems when I follow your command image

Connum commented 1 month ago

For anyone coming across this in 2024: WHISPER_CUBLAS is deprecated, you have to use GGML_CUDA instead. If you keep getting the deprecation message, delete your build location first. So in order to build directly from the checked-out repository into a /dist folder, you'd use the following:

It might have to do with me using only the Build Tools instead of the full VS installation, but I also had to copy the files from C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.6\extras\visual_studio_integration\MSBuildExtensions to C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\MSBuild\Microsoft\VC\v170\BuildCustomizations for the build configuration to run through.

Also make sure that your CUDA_PATH environment variable is set correctly.