[Tracker] General feature requests for torchao

[supriyar]

This issue tracks outstanding feature requests for torchao. If you'd like a specific feature to be added to torchao, please comment directly here.

Quantization Techniques (based on planned, new requests)

• [ ] GPTQ
• [ ] HQQ

DTypes

• [ ] fp8
• [ ] mx format

Sparsity APIs

• [ ] int8 + 2:4 sparsity
• [ ] fp8 + 2:4 sparsity

Kernels

• [ ] kernel autotuner for dynamic quant
• [ ] C++ extension for custom kernel - starting with PagedAttention (https://github.com/pytorch/pytorch/issues/121465#)
• [ ] CUTLASS w4a8 kernel https://github.com/pytorch-labs/ao/issues/64

cc @cpuhrsch

[mergennachin]

GGUF conversion and kernels

• [ ] Add gguf conversion scripts using gpt-fast/model.py as source
• [ ] Add torch subclass to ao that does the quantization
• [ ] Add ggml kernel with the wrapper to make sure it executes in eager but calls the specific kernel
• [ ] Reference python implementation

cc @larryliu0820