Any tips or suggestions on how we could generate a faster quantization, at the cost of accuracy, and still remain EXL2-compatible? Is a single pass possible, maybe at the cost of not exactly hitting the target avg. bit depth? Would skipping activation-aware quantization be worth it for 6 or 8 bit?
load-in-8-bit works reasonably on-the-fly, I'm looking for some way to re-create something like that, but runnable with Exllama.
One application is testing intermediate CPs while training. Would running measurement on the base model, then quantizing later checkpoints as we fine-tune without re-measuring perform reasonably, egs., at 6 or 8 bits to minimize the error?
Solution
Just a discussion is fine, maybe we don't need API changes.
Alternatives
No response
Explanation
EXL2 quantization of larger models can take a really long time.
Why EXL2 compatible? Exllama has wider compatibility with consumer hardware than other engines (egs., vLLM), and better efficiency than transformers, including TP support, and is faster than GGUF with GPUs.
Examples
No response
Additional context
No response
Acknowledgements
[X] I have looked for similar requests before submitting this one.
[X] I understand that the developers have lives and my issue will be answered when possible.
[X] I understand the developers of this program are human, and I will make my requests politely.
Problem
Any tips or suggestions on how we could generate a faster quantization, at the cost of accuracy, and still remain EXL2-compatible? Is a single pass possible, maybe at the cost of not exactly hitting the target avg. bit depth? Would skipping activation-aware quantization be worth it for 6 or 8 bit?
load-in-8-bitworks reasonably on-the-fly, I'm looking for some way to re-create something like that, but runnable with Exllama.One application is testing intermediate CPs while training. Would running measurement on the base model, then quantizing later checkpoints as we fine-tune without re-measuring perform reasonably, egs., at 6 or 8 bits to minimize the error?
Solution
Just a discussion is fine, maybe we don't need API changes.
Alternatives
No response
Explanation
EXL2 quantization of larger models can take a really long time.
Why EXL2 compatible? Exllama has wider compatibility with consumer hardware than other engines (egs., vLLM), and better efficiency than transformers, including TP support, and is faster than GGUF with GPUs.
Examples
No response
Additional context
No response
Acknowledgements