Closed madhavatreplit closed 1 year ago
We had an open source PR on the HF repo that enables 8-bit and 4-bit quantization: https://huggingface.co/replit/replit-code-v1-3b/discussions/19/files
Replicated those PR's changes here.
Tested that loading in 8-bit and 4-bit quantization work.
Why
We had an open source PR on the HF repo that enables 8-bit and 4-bit quantization: https://huggingface.co/replit/replit-code-v1-3b/discussions/19/files
What changed
Replicated those PR's changes here.
Testing
Tested that loading in 8-bit and 4-bit quantization work.
Rollout