This structure allows us to swap out modules like QuarotFP16* modules (which emulate INTX quantization in FP16) for real ones which use CUDA kernels, like Linear4bit.
It also gets rid of any monkeypatching business and is overall much clearer to model implementers, though for now this requires a fair bit of implementation.
This structure allows us to swap out modules like QuarotFP16* modules (which emulate INTX quantization in FP16) for real ones which use CUDA kernels, like Linear4bit.
It also gets rid of any monkeypatching business and is overall much clearer to model implementers, though for now this requires a fair bit of implementation.