Closed eswarthammana closed 5 months ago
k-quants has something a bit similar: https://github.com/ggerganov/llama.cpp/blob/2046eb4345e62c4575b3cdc0115a51db89f3fb70/llama.cpp#L8135
It quantizes some more important layers/tensors less. It's a tuned preset to increase the quantization quality, not a dynamic thing determined at runtime though.
Thanks for the update, then shall we consider llama.cpp provides weight quantization and activation quantization (layers)
shall we consider llama.cpp provides weight quantization and activation quantization (layers)
I'm not completely sure what you mean. Most tensors get quantized, the main exception is 1-dimensional tensors which are too small to benefit much. (Also I think that's partially for performance reasons.)
Just for example, here's output from loading a Q5_K_M quantized Mistral model (7B):
Thanks for the detailed explanation :)
This issue was closed because it has been inactive for 14 days since being marked as stale.
Hi i need some information. could you please respond.
Does llama.cpp uses dynamic range calibration.