Bug: imatrix quant gguf models (e.g. IQ3_XS, IQ2_M) not using NV GPU properly with `llamafile-0.8.14`

[wingenlit]

Contact Details

No response

What happened?

when using important matrix quantized models on llamafile, with model fully offloaded to GPU, it appears that GPU is not doing much compute, while CPU is under heavy load.

notes:

• recompile feature is used; and successful
• model is fully offload with nvtop confirmation on vram.
• tested with same command on v0.8.13 v0.8.9 v0.8.6 without the same problem
• seen strong degradation in performance on nvidia gpu (RTX4070 sm_89) on v0.8.12 and up

Version

llamafile v0.8.14

What operating system are you seeing the problem on?

Linux, Windows

Relevant log output

No response

[jart]

We've needed to disable the IQ quants by default in GGML CUDA for the time being, due to issues surrounding code size and compile times. We're hoping to find a better workaround soon. But in the meantime, you can fix this by passing --recompile --iq to llamafile (assuming you've built it at HEAD) which will compile a ggml-cuda module for your system that has support for IQ.