search

analogdevicesinc / ai8x-synthesis

Quantization and Synthesis (Device Specific Code Generation) for ADI's MAX78000 and MAX78002 Edge AI Devices
Apache License 2.0
55 stars 47 forks source link

Bad Evaluation after training new KWS-Model #231

Closed raphex24 closed 2 years ago

raphex24 commented 2 years ago

Hello, i wanted to compare results of KWS with and without MFCC Calculation. So i changed the KWS20_v3 model so that it directly supports MFCC calculations. What i did was to change the __gen_datasets in KWS20 so that it generates (16000x1) instead of (128x128). And during training the getitem function converts inp into a numpy, calculates MFCCs and transforms back into torch.

For the new model, i wrote the corresponding yaml and generated an example .npy-file (for ai8xize.py). I used all the standard scripts of kws20_v3 for trianing evaluation and quantization and just fitted the path to the new model file and the model name.

During training i get around 60% - 80% from evaluating the training results (depending on choosen layers). But after using quantize.py the evaluation will drop below 10%.

Usually i don't expect a model to drop that much after quantization. Might there be any other files i have to change?

And a second question: It's said that the linear layer cannot be larger than 1023. A model with an layer of a size around 800 was still causing troubles. The ai8xize.py throw an error, because the layersize is -3. After changing it to a size below 255 (more CNNs before) it was working fine. So is that true, that i can't have larger layersizes if i use quantisation?

aniktash commented 2 years ago

Hi, The first problem is most probably related to normalization. The dataloader calls ai8x.normalize() function to normalize the data in [-128/128,127/128] for training and [-128,127] for evaluating quantized result. Please check "Data Loader" section on "Normalizing Input Data" in readme file for more info. For the second question, the input limit is 16*64=1024. It could be a mismatch in yaml or some other limits. Quantization should not have any impact on that.

raphex24 commented 2 years ago

Thanks for the respond. The ai8x.normalize() seems to solve the issue. However i was using my own normalization and quantisation before, as i added a further clipping to the normalization. So i thought i wouldn't need to do it a second time.