search

Cornell-RelaxML / QuIP

Code for paper: "QuIP: 2-Bit Quantization of Large Language Models With Guarantees"
338 stars 31 forks source link

will it support group quant #14

Open oreo0906 opened 1 month ago

oreo0906 commented 1 month ago

I confused the method ldlq_Rg dont support group quantization.

jerry-chee commented 1 month ago

we did not investigate group quantization in this codebase, which is why it is not supported. but the method is compatible with group quantization. but incoherence and grouping aim to do similar things. Incoherence reduces overall variation in the weights via a random orthogonal matmul, while grouping reduces the effect of variation in the weights when quantizing by adding additional scaling params.