Open Interpause opened 2 months ago
Description
HuggingFace's Quanto has implemented 4 bit & 2 bit KV cache quantization compatible with Transformers. See: https://huggingface.co/blog/kv-cache-quantization
I may PR when I've time to experiment.
Seems definitely possible: https://github.com/Vahe1994/AQLM/issues/85#issuecomment-2194691934
But man they made me project lead for something in uni so im in a time crunch
Description
HuggingFace's Quanto has implemented 4 bit & 2 bit KV cache quantization compatible with Transformers. See: https://huggingface.co/blog/kv-cache-quantization
I may PR when I've time to experiment.