Unable to use GPU acceleration

Maknee / minigpt4.cpp

Port of MiniGPT4 in C++ (4bit, 5bit, 6bit, 8bit, 16bit CPU inference with GGML)

MIT License

549 stars 26 forks source link

Unable to use GPU acceleration #21

Open deadpipe opened 8 months ago

deadpipe commented 8 months ago

@Maknee

I tried setting option(MINIGPT4_CUBLAS "minigpt4: use cuBLAS" ON) in the CMakeLists.txt.

But when i run cmake --build . --config Release,

i get this error below unfortunately : -

C__Windows_System32_cmd exe 24_11_2023 00_27_05

Any advice to deal with is highly appreciated

Originally posted by @deadpipe in https://github.com/Maknee/minigpt4.cpp/issues/16#issuecomment-1824829662

Maknee commented 8 months ago

Sorry it hasn't been documented, but I have not tested cublas integration with minigpt4

deadpipe commented 8 months ago

@Maknee Do you have this on your roadmap to fix this CuBlas integration with minigpt4.cpp?

Maknee commented 7 months ago

Unfortunately I don't have that much time these days -- I believe that cublas could be integrated nicely just with the vicuna model from llama.cpp, which should provide a nice perf boost