hseok-oh
commented
3 months ago Below is schema draft to represent ggml's quantization type (block quntization) Please give your opinion about this. @seanshpark @mhs4670go @jinevening @chunseoklee @glistening @jyoungyun @ragmani
// Block quantization: from ggml quantization (https://github.com/ggerganov/ggml)
table CircleBlockQuantization {
name:string;
}
// Represents a specific quantization technique's parameters.
union QuantizationDetails {
CustomQuantization,
CircleBlockQuantization
}
// Parameters for converting a quantized tensor back to float.
table QuantizationParameters {
// These four parameters are the asymmetric linear quantization parameters.
// Given a quantized value q, the corresponding float value f should be:
// f = scale * (q - zero_point)
// For other quantization types, the QuantizationDetails below is used.
// NOTE min/max values are valid if
// 1. length of min/max == 0 or
// 2. length of min/max == length of scale/zero_point
// Otherwise, min/max are not valid (undefined behavior).
min:[float];
max:[float];
scale:[float]; // For dequantizing the tensor's values.
zero_point:[long];
// If this is not none, the other quantization parameters (i.e. min, max,
// scale, zero_point fields above) are ignored and the value of the
// QuantizationDetails union should be used.
details:QuantizationDetails;
// Specifies the dimension of the Tensor's shape that the scales and
// zero_points correspond to. For example, a tensor t, with dims=[4, 3, 2, 1]
// with quantization params:
// scale=[1.0, 2.0, 3.0], zero_point=[1, 2, 3], quantization_dimension=1
// will be quantized across the second dimension of t.
// t[:, 0, :, :] will have scale[0]=1.0, zero_point[0]=1
// t[:, 1, :, :] will have scale[1]=2.0, zero_point[0]=2
// t[:, 2, :, :] will have scale[2]=3.0, zero_point[0]=3
quantized_dimension:int;
}It introduces new QuantizationDetails's union table CircleBlockQuantization for detail field. details field is never used yet, so it will become first usage. CircleBlockQuantization has name field and it has ggml's quantization type name (ex. Q4_0, Q4_1, Q8_0, etc). If details field have any value, other QuantizationParameters field will not be used to decide quantization type.
Quantization parameters such as scales are in buffer with quantized value, so there is no field to save deltas (scales) for each block, and it is same policy with ggml's quantization - dequantization.
Below is Q4_0 type block structure in buffer.
https://github.com/ggerganov/ggml/blob/2438d62cb9290b5b5dc6228dec76fe81cf64238e/src/ggml-common.h#L144-L149
#define QK4_0 32
typedef struct {
ggml_half d; // delta
uint8_t qs[QK4_0 / 2]; // nibbles / quants
} block_q4_0;
static_assert(sizeof(block_q4_0) == sizeof(ggml_half) + QK4_0 / 2, "wrong q4_0 block size/padding");(ggml_half: fp16)
Addition: @glistening 's comment https://github.com/Samsung/ONE/pull/13693#discussion_r1732497335
Is the prefix
Circlenecessary to avoid name conflict from flatbuffers generated files? I guessGGMLBlockQuantizationmay be better as @jinevening suggested offline. It makes it clear whatCircleBlockQuantizationmeans.
jinevening
seanshpark
glistening
What?
Let's support block quantization data type on circle format to support LLM model.
Why?
To support LLM model, we need to support small size weight quantization with small precision loss. So we need to introduce chunk quantization such as ggml (llama.cpp) 's quantization type.
To represent this, we need to expand circle schema's
QuantizationParameterstable or/andQuantizationDetailsunion.Related issue: #13742