AndreaLombax
commented
7 months ago Same problem here. No updates?
EDIT 1:
By following this post:
I checked my CUDA version using the nvidia-smi command, and it was CUDA 12.0. My libcudart version on the machine was, instead, 11.5 (you can check by running the following command apt search cuda-toolkit.
I then just downloaded and installed the corresponding cudatoolkit version by following the official nvidia website and rebooted the machine. It now works.
EDIT 2:
I also had to update nvidia drivers to match CUDA and cudatoolkit versions.
I have executed the following commands:
sudo apt-get install -y nvidia-kernel-open-535
sudo apt-get install -y cuda-drivers-535
My nvidia-smi looks like:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03 Driver Version: 535.129.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A16 On | 00000000:0B:00.0 Off | 0 |
| 0% 31C P0 25W / 62W | 3198MiB / 15356MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA A16 On | 00000000:14:00.0 Off | 0 |
| 0% 31C P0 25W / 62W | 3052MiB / 15356MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
I just used base script from the TheBloke repo and got an error during loading the model:
OSError: libcudart.so.12: cannot open shared object file: No such file or directory
my script is : from ctransformers import AutoModelForCausalLM llm = AutoModelForCausalLM.from_pretrained("TheBloke/CodeLlama-7B-GGUF", model_file="codellama-7b.Q4_0.gguf", model_type="llama", gpu_layers=50)
print(llm("AI is going to"))