Open johnhaire89 opened 2 months ago
@Van-QA @imtuyethan I think that is something we implemented regarding error handling in the past? Which leads the user to the CUDA additional installation page.
@johnhaire89 FYI, Jan is in the process of overhauling how we deal with llama.cpp binaries and GPU dependencies.
llama.cpp
now bundles its CUDA dependenciesllama.cpp
engine@Van-QA I will keep this bug open. Once we clean up PM systems, let's link the 2 epics that would solve this bug. My style is to only close bugs once the corresponding feature is shipped.
cortex engines llama.cpp install
should also pull CUDA dependences, cc @namchuai (FYI)Handling this bug as part of https://github.com/janhq/cortex.cpp/issues/1165
Hi @dan-homebrew @imtuyethan. This is a known issue, there is a fix in 0.5.4: https://github.com/janhq/jan/issues/3552.
We have this step to let user install additional dependencies right in the app (without redirecting users out of the app).
In the next update of integrating cortex-cpp
engine pull, there should be no extra request to install these dependencies, BUT there this error message would really help in case there is a Driver/Cuda update that does not work with the pulled engine & it's dependencies.
The fix is included in Jan's path to cortex.cpp: https://github.com/janhq/jan/issues/3690
Current behavior
I was playing with Jan for the first time and realised that GPU acceleration wasn't enabled. I toggled the "GPU Acceleration" switch to enable it for my NVIDIA RTA A2000 with no error.
When I next typed into the chat window, Jan wasn't able to start the model.
Problem was that I didn't have CUDA toolkit installed. Per SO answer at https://stackoverflow.com/a/55717476,
nvidia-smi
shows the supported CUDA version, butnvcc --version
should be used to check the installed version. I installed CUDA Toolkit and it's back to working like magic.This is probably more a feature request then a bug, but that toggle should probably show an error if I try to enable GPU acceleration for a nvidia card when CUDA toolkit isn't installed.
Minimum reproduction step
Start with a Windows PC with a NVidia gpu and CUDA Toolkit not installed (per
nvcc --version
)Expected behavior
When I try to enable GPU Acceleration for a Nvidia GPU in an environment where CUDA Toolkit isn't installed, I should get a helpful error. Maybe a warning can be displayed next to GPU in the dropdown?
Screenshots / Logs
Jan version
0.5.2
In which operating systems have you tested?
Environment details
Windows 11 NVIDIA RTX A2000 8GB Laptop GPU8192MB VRAM CUDA toolkit not installed