TIDL quantize

[happy-Bobo]

hi Manu, Do you know the TIDL quantizatation which support by TI's tool——tidl_model_import.out.exe? would you please tell me about it's different with it caffe-jacinto. thanks best regards

[mathmanu]

Hi, it's similar, but not exactly same.

Caffe-jacito does not do quantitied inference, it only simulates quantization with floating point operations - and the simulation is not perfect. However, I hope it helps to give you and idea about the accuracy that you can get with quantitied inference.

[happy-Bobo]

Hi Manu， thanks for your reply, the accuracy loss just a little when I use the caffe-jacinto to test quantize, but when I use the TIDL do inference on platform device, the accuracy loss much whatever 8 bit or 12 bit quantize, the feature_map which I get on platform device is the integer of 0-256, it is not a signed float vector. what's more, I am used the cosine to judge the feature_map of two pictures is belong to same class or not, it is not influence the cosine result even I convert the feature_map to float vector by divide the scale factor. so I am confused. please help me! thanks best regards

[mathmanu]

CC: @kumardesappan 8 bit quantized values are supposed to be in the range 0-255 or -128-127 depending on whether they are unsigned or signed respectively. This is a quantized representation of the corresponding floating point value. What is your confusion?

[happy-Bobo]

Hi manu thanks for your reply, it is same with you said, I use the "signed char* " to point the output buffer, so the feature I got is -128 to 128, and then ,I got the features of 10000 images by TIDL, I used the features to test the accuracy of my model, it loss 5% than before the quantize. But when I use the Caffe-jacito do quantitied inference on pc, the accuracy only loss 1%. Just as I describe in the follow post： https://e2e.ti.com/support/processors/f/791/t/760646

thanks best regards Bobo

[mathmanu]

I am not able to give a solid answer since this issue is not related to caffe-jacinto. However, let me try to understand if your hyper parameters and see if they can be improved. Can you share your solver prototxt?

[happy-Bobo]

Hi manu, thank you so much！my net is almost same with imagenet, I just do some fine tune on it for train a face classification model. there are all the files for train and inference : deploy.prototxt.txt run.log solver.prototxt.txt test.prototxt.txt train.prototxt.txt thanks best regards Bobo

[mathmanu]

I checked your train.prototxt. You seem to have 10575 output classes. This is too many to give good results with 8-bit quantization.

You may have to increase the bitwidth for activations. If not for all the all layers, at-least for the fully connected layers.

Another option may be to reduce the number of classes - by some kind of tree structured hierarchical classification.

[happy-Bobo]

thanks for your help, the net does have 10575 output classes, but the fully connected layer is not contain in the deploy.prototxt when I do the quantize. does it still influence the result？

[mathmanu]

You mean the fully connected layers were run outside the network? Was it in floating point?

Anything can be suspected in this case since there are so many classes - including the numerical precision of your implementation.

[happy-Bobo]

No, I don't need the output of last fully connected layer, I just need the output of first fully connected layer, this vector represent the feature of my input image, and then, I use the cosine angle of two vector to judge them are belong to same class or not.

[mathmanu]

I checked your features in deploy.prototxt - it just has 512 outputs. I think more output channels will help if you have so many classes.

Another thing to note is that setting quantize:true does not enabble quantization for the first frame. But from second frame input onwards, quantization will happen if you use the same Net() object for inference.

Please note that quantization will be applied only from second inference onwards: https://github.com/tidsp/caffe-jacinto/blob/caffe-0.17/src/caffe/proto/caffe.proto#L2398

[happy-Bobo]

thanks so much for your suggestion, I will try to add the output to 1024 and do some test. I know that quantization dose not enable for the first frame. But I have another question, I got the feature vector of 23 pictures with quantize:true. test_quantize_pc.txt And then, I got the feature vector of the same 23 pictures by TIDL. test_quantize_TIDL.txt them are represent the same pictures,one is float and another is signed char, but I don't find the relation of them. Such as the feature of the last picture. -0.711335 -0.711335 22.0514 32.0101 ...... 1 -22 35 61 ......

thanks best regards

[mathmanu]

Quantized values (i.e. integer /char/sighned char values) are equivalent to multiplying the float value by a suitable scale factor and then rounding to the nearest integer. So if you compare two tensors closely you can find a scale factor that converts from the floating point value to the inter values.