search

huggingface / local-gemma

Gemma 2 optimized for your local machine.
Apache License 2.0
345 stars 27 forks source link

8bit quantization #30

Open paolo-losi opened 4 months ago

paolo-losi commented 4 months ago

Would it be possible to support 8bit quantization?

SunMarc commented 4 months ago

Hi @paolo-losi, we try to keep the number of args low. Hence we decided to go with 4bit quantization with the memory preset. Is there an issue with the quality of the 4-bit model ?