[Tracker] WIP features for torchao 0.4 - Githubissues

pytorch / ao

PyTorch native quantization and sparsity for training and inference

BSD 3-Clause "New" or "Revised" License

1.64k stars 182 forks source link

[Tracker] WIP features for torchao 0.4 #493

Closed jerryzh168 closed 3 months ago

jerryzh168 commented 4 months ago

Release date: Aug 8 2024 Branch cut: Aug 2 2024

Developer Facing API

[x] static quantization flow example @jerryzh168
[ ] QAT refactor to generalize to other dtypes/techniques @andrewor14
[x] Int4 weight-only QAT

Developer Facing API use cases

[ ] hqq, hqq-mix subclasses (defer to next release) @HDCharles
[x] AffineQuantizedTensor layout cleanup @jerryzh168
[x] [postponed to 0.5] int4 weight only quantization change device support (e.g. cpu -> cuda) @jerryzh168
[x] sparse + quantization composability support @jcaip
[x] quantize kv_cache to int8 for gpt-fast/torchAO llama @HDCharles

Modeling user API

[ ] autoquant to use AffineQuantizedTensor @HDCharles
[x] [handed off to executorch team] torchchat/executorch compatibility @jerryzh168
[x] add sam-fast to torchao @jcaip

Documentation

[ ] add quantization overview to torchao doc @supriyar
[ ] Huggingface neural-magic SparseLlama 2:4 notebook

supriyar commented 3 months ago

tracker for 0.5 is here https://github.com/pytorch/ao/issues/667