search

qwopqwop200 / GPTQ-for-LLaMa

4 bits quantization of LLaMA using GPTQ
Apache License 2.0
3k stars 458 forks source link

6-bit quantization #236

Open philipturner opened 1 year ago

philipturner commented 1 year ago

For smaller models, quantization causes more quality loss than large models. Could the repository try 6-bit / 128 groups for stuff like LLaMa-7B? This could be most useful for some of the smaller language networks in Stable Diffusion.

Ph0rk0z commented 1 year ago

Yes.. 6b would work great for 13b and below to make the model smarter.