Re: Clarification on conversion/quantization of Floating Point Numbers to 2-bit Binary

[Sethu11]

Let's take JSC-S in particular, so the input bit-width as well as the output bit-width is 2. So, how do you convert or rather quantize the Floating Point Numbers in the dataset into 2-bit Binary numbers to feed them as inputs. (Image Attached for further clarification).

How do you convert/quantize FP Numbers on the left side to Binary Numbers on the right side?

[nickfraser]

(Image Attached for further clarification).

How have you generated these files?

So, how do you convert or rather quantise the Floating Point Numbers in the dataset into 2-bit Binary numbers to feed them as inputs.

What exactly do you mean by "how"? There are many ways to answer this question. Do you want to know where the code is? Do you want to understand the mathematical expression?

During training, we leverage the quantisers from brevitas to learn a quantizer which is optimised for the problem at hand. So the parameters of the input quantization are a property of the model themselves, but effectively, each model performs a uniform-spaced quantization (to integers) equivalent to the formula:

x_quant = clip(round(x/s + b), -2, 1)

where x is the input vector and s / b are vectors of learnt scales / biases for the dataset during training.

[Sethu11]

The left side file (in the ref. image) I saved the entire numpy array onto a .txt file from 'dataset.py' after you save the input in 'X_train_val' and output in 'y_train_val' respectively. As to the right one, I ran 'dataset_dump.py' and got the input and output file. I just wanna know the mathematics behind the conversion, like how are you converting a FP Number into a 2-bit binary, that info is sufficient for me. If you can point me to the code for it, that's even better.

[nickfraser]

The module which learns the quantization parameters is the QuantBrevitasActivation which is a child module (module.input_quant) of the first SparseLinearNeq in the nn.ModuleList in JetSubstructureNeqModel. Probably the best place to look is in the dataset_dump.py script. input_quant is called with bin_output set, so the forward function is here.

[Sethu11]

Is there any way by which I can see the values of scales(s) and biases(b) for the particular dataset (JSC-S/M/L).

[nickfraser]

They need to be calculated from all the modules in the pre_transform list (BatchNorm & ScalarScaleBias). You can look at their parameters directly and from there calculate s & b. They are not pre-calculated anywhere.