Closed bartowski1182 closed 1 year ago
Are you running the exe? Building from source (did you run make clean)? Windows or linux?
Good questions, building from source, clean make, tried with both make LLAMA_CUBLAS=1 and make LLAMA_OPENBLAS=1 LLAMA_CLBLAST=1, Ubuntu 22.04
It's not even attempting to load anything onto the GPU, if i watch it with nvidia-smi it stays at 1mb
sigh ignore me, i was using the wrong image.. spent a solid 30 minutes debugging a typo, love it
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
Loading a 7B q2_K model should easily fit all layers into my 3060 with 12GB of VRAM
Current Behavior
Even loading with only 20 layers of 7B q2_K offloaded results in an error saying i'm out of VRAM
Environment and Context
Ryzen 3600 64GB ram, 3060 12GB RAM
Steps to Reproduce
Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.
Failure Logs