Max191
commented
12 months ago I created some linalg IR to represent this reassociation so we can have something to look at and start iterating on https://gist.github.com/Max191/c76474d1d484cb05d191543877110d28
One thing to note that is slightly different from what it seems GGML is doing in computing the scaled sums of the vector:
%vec_scaled_sums = linalg.generic {indexing_maps = [affine_map<(d0, d1, d2, d3) -> (d0, d1, d2, d3)>,
affine_map<(d0, d1, d2, d3) -> (d0, d1, d2)>],
iterator_types = ["parallel", "parallel", "parallel", "reduction"]}
ins(%vector : tensor<1x1x32x128xf32>)
outs(%2 : tensor<1x1x32xf32>) {
^bb0(%vec: f32, %out: f32):
%sum = arith.addf %vec, %out : f32
linalg.yield %sum : f32
} -> tensor<1x1x32xf32>Here I simply use the sum of the original unquantized values. However GGML first quantizes, then computes the sum of the quantized values times the scales:
static void quantize_row_q8_1_reference(const float * restrict x, block_q8_1 * restrict y, int k) {
assert(QK8_1 == 32);
assert(k % QK8_1 == 0);
const int nb = k / QK8_1;
for (int i = 0; i < nb; i++) {
float amax = 0.0f; // absolute max
for (int j = 0; j < QK8_1; j++) {
const float v = x[i*QK8_1 + j];
amax = MAX(amax, fabsf(v));
}
const float d = amax / ((1 << 7) - 1);
const float id = d ? 1.0f/d : 0.0f;
y[i].d = d;
int sum = 0;
for (int j = 0; j < QK8_1/2; ++j) {
// GGML quantizing values here
const float v0 = x[i*QK8_1 + j]*id;
const float v1 = x[i*QK8_1 + QK8_1/2 + j]*id;
y[i].qs[ j] = roundf(v0);
y[i].qs[QK8_1/2 + j] = roundf(v1);
// Then computing the sum of the quantized values
sum += y[i].qs[ j];
sum += y[i].qs[QK8_1/2 + j];
}
// Multiply by scales
y[i].s = sum*d;
}
}The GGML implementation loses some information here, and from my testing, the results of the matmul are more precise when computing the scaled sum before quantization rather than after quantization.
bjacob
MaheshRavishankar
This issue is to provide some context and discussion for current quantized matmuls compared against what is done for GGML in llama.cpp.
So far, I have looked at the
i4quantization case. In llama2, we have been looking at matvecs, where the corresponding inputs to GGMLs matmul kernel would be a LHS quantized matrix times a RHS unquantized vector.The matmul is defined in ggml_compute_forward_mul_mat, and the functions for handling vector dot products for various quantization types are defined here
The critical code is in how it handles these vector dot products:
GGML dequantizes the RHS here into the vec_dot_type (
GGML_TYPE_Q8_1) before performing the matmul, and then usesggml_vec_dot_q4_1_q8_1for the matmul.The dot product looks like this:
The sum of products
sumiis computed ini8, with thei4LHS being unpacked here into byte sized containers, and thensumiis multiplied by the scales for the LHSx[i].dand RHSy[i].d. Then they handle the zero points after the sum of products by addingGGML_FP16_TO_FP32(x[i].m)*y[i].sx[i].mis the min value of the LHS quantized group, whiley[i].sis equal tosum_j(y[i].qs[j] * y[i].d)(aka the scale times the sum of the quantized weights. This is computed inquantize_row_q8_1).So the rhs is symmetrically quantized dynamically, and then the dot product is reassociated from:
to
It is also worth noting that GGML does
((lhs_q*lhs_scale)+lhs_zp)and we do((lhs_q-lhs_zp)*lhs_scale), so I think they save 1 multiplication in the reassociated form with this quantization difference.The quantization scheme is essentially the same, however, and the main thing that GGML does to do these multiplications in
i8is the dynamic requantization of the RHS vector back intoi8before doing the matmul/matvec.