QuaRot bugfixes - Githubissues

microsoft / TransformerCompression

For releasing code related to compression methods for transformers, accompanying our publications

MIT License

354 stars 31 forks source link

Closed nailimixaM closed 3 months ago

nailimixaM commented 3 months ago

GPU OOM fix: flash attention forward pass should not actually do KV caching, we emulate it by quantizing and immediately dequantizing the Ks and Vs.
Various PPL mismatches: missing symmetric arg so always True by default caused some issues, and similar bugs

Added tests and lots of refactoring to GPTQ.