Support quantization beyond 4 bits

instructlab / training

InstructLab Training Library - Efficient Fine-Tuning with Message-Format Data

https://pypi.org/project/instructlab-training/

Apache License 2.0

21 stars 45 forks source link

Support quantization beyond 4 bits #29

Open RobotSail opened 5 months ago

RobotSail commented 5 months ago

We currently only support 4-bit quantization via BitsAndBytes. We should support other options such as 8-bit, (potentially) 6-bit, etc.

github-actions[bot] commented 3 days ago

This issue has been automatically marked as stale because it has not had activity within 90 days. It will be automatically closed if no further activity occurs within 30 days.