Disparity between keras/tflite version and Kmodel

[krishnak]

This is similar to issue 114 @sunnycase @zye1996 I am facing similar issues, that is, the Keras model detection and Kmodel detections are so far apart.

I tried using the CI version for Linux,

what does --weights-quantize-threshold do? I had two warnings against 2 Conv2D layers(these layers were recently added), I used a value of 64 which removed one warning, I changed it to 128 which removed both the warnings.

However the resultant model is still performing poorly as the previous kmodel compiled using ncc (which doesn't have the weights-quantize-threshold option)

Does this issue occur only when there is Conv2D operation? I recently modified my model to have 2 extra layers of Conv2D - all other layers are separable Conv2D.

Inspite of using CI version, the KModel produced detects only 50% of the objects.

Earlier I had a model only made of separable Conv2D layers, the performance of the previous model was similar on Keras and on Kmodel. It used to detect 98% in most cases

I added 2 extra layers of Conv2D to model made of serparable Conv2D layers which seems to alter things completely.

I also tried using inference type =float when creating Kmodel, the resultant file was 1.4MB and the application core dumped on K210 saying not enough memory.

How do I optimise the Keras model to Kmodel?

[zye1996]

This is similar to issue 114 @sunnycase @zye1996 I am facing similar issues, that is, the Keras model detection and Kmodel detections are so far apart.

I tried using the CI version for Linux,

what does --weights-quantize-threshold do? I had two warnings against 2 Conv2D layers(these layers were recently added), I used a value of 64 which removed one warning, I changed it to 128 which removed both the warnings.

However the resultant model is still performing poorly as the previous kmodel compiled using ncc (which doesn't have the weights-quantize-threshold option)

Does this issue occur only when there is Conv2D operation? I recently modified my model to have 2 extra layers of Conv2D - all other layers are separable Conv2D.

Inspite of using CI version, the KModel produced detects only 50% of the objects.

Earlier I had a model only made of separable Conv2D layers, the performance of the previous model was similar on Keras and on Kmodel. It used to detect 98% in most cases

I added 2 extra layers of Conv2D to model made of serparable Conv2D layers which seems to alter things completely.

I also tried using inference type =float when creating Kmodel, the resultant file was 1.4MB and the application core dumped on K210 saying not enough memory.

How do I optimise the Keras model to Kmodel?

I did not find a solution to the model when deployed on the k210. But when I tried to run ncc infer it does generate the correct result on my PC side, see issue 116

I think --weights-quantize-threshold set the range limit of output in each layer under which the weights of that layer will be quantized, otherwise calculation is in float. But I don't know how k210 chip is handle this mechanism since it is new?

Can you try run the model on the computer and examine the output to see whether it matches expectation?

[krishnak]

@zye1996 How to use the bin file produced byncc infer command? Is there any documentation?

My initial model when run on PC using keras and after conversion to KModel, produced almost identical results. They are as below

https://drive.google.com/open?id=1G4tW21Pl7LyZAlYiz46IezRwylPUq5_e https://drive.google.com/open?id=1FvAVnkQhiLfu-MAcoz3NnyAG-_lyl4Jx https://drive.google.com/open?id=1GFMpfmarFMvcLnQFbOZJhcrM2YZ2au0R

My latest model has 2 extra layers of Conv2D, this one is producing far greater results on my PC compared to my earlier model (when using Keras), but when converted to Kmodel, it is NOT detecting anything for the images which have large faces, for the tiny faces it is detecting very few around 7, it used to detect around 97 with my earlier KModel.

https://drive.google.com/open?id=1FKx_Fv8M3MJnY_EN58jimRhNr61QDF7G

These conversions are with an earlier version of ncc NOT the latest one - so this issue is NOT a new issue related to the version of ncc.

@sunnycase this could also be an issue at the time of keras model getting converted to tensorflow lite, because tensorflow lite converter keeps changing the output order without any change done on the model's output.

I will try to see whether the tensorflow lite model produces the desired result on the PC to identify where the issue is

@zye1996 Regarding your model, I will post a question there to keep matters clear.

[krishnak]

I have just tested the tflite file, the inference on tflite model and keras model are identical. So we can ignore tflite being the culprit.

[krishnak]

I have run the following ~/nncase/ncc compile mymodel.tflite mymodel.kmodel -i tflite --dataset ~/nncase/images/

he above kmodel detects only 7 out of 96

This following detects 22 objects out of 96 of the group photo in the previous message

~/nncase/ncc compile mymodel.tflite mymodel.kmodel -i tflite --dataset ~/nncase/images/ --calibrate-method l2

However when I test it on this image, which is an easy detection, both the models don't detect the single face in the frame

@sunnycase any suggestions

[zye1996]

@zye1996 How to use the bin file produced byncc infer command? Is there any documentation?

My initial model when run on PC using keras and after conversion to KModel, produced almost identical results. They are as below

https://drive.google.com/open?id=1G4tW21Pl7LyZAlYiz46IezRwylPUq5_e https://drive.google.com/open?id=1FvAVnkQhiLfu-MAcoz3NnyAG-_lyl4Jx https://drive.google.com/open?id=1GFMpfmarFMvcLnQFbOZJhcrM2YZ2au0R

My latest model has 2 extra layers of Conv2D, this one is producing far greater results on my PC compared to my earlier model (when using Keras), but when converted to Kmodel, it is NOT detecting anything for the images which have large faces, for the tiny faces it is detecting very few around 7, it used to detect around 97 with my earlier KModel.

https://drive.google.com/open?id=1FKx_Fv8M3MJnY_EN58jimRhNr61QDF7G

These conversions are with an earlier version of ncc NOT the latest one - so this issue is NOT a new issue related to the version of ncc.

@sunnycase this could also be an issue at the time of keras model getting converted to tensorflow lite, because tensorflow lite converter keeps changing the output order without any change done on the model's output.

I will try to see whether the tensorflow lite model produces the desired result on the PC to identify where the issue is

@zye1996 Regarding your model, I will post a question there to keep matters clear.

To use the .bin file, simply load the file with numpy load function, note the data type as np.float32. And you can then slice the data into different feature maps and reshape them into correct shapes. The original data format is in CHW sequence so you may need to tranpose each feature map to HWC for decoding.

[krishnak]

I am doing this

>>> import numpy as np
>>> a = np.fromfile("5.bin")
>>> print(a.shape)
(33075,)

but my kmodel should be outputing 7x10x45 + 14x20x45 + 28x40x45 = 66150

I only get half the data? am I missing something

[zye1996]

I am doing this

>>> import numpy as np
>>> a = np.fromfile("5.bin")
>>> print(a.shape)
(33075,)

but my kmodel should be outputing 7x10x45 + 14x20x45 + 28x40x45 = 66150

I only get half the data? am I missing something

You need to do a=np.fromfile("5.bin", dtype=np.float32)

[krishnak]

@zye1996

ncc is version 0.2

Thanks for the tip, I have got interesting results

for the group photo as above

I am getting 99 detections with the ncc infer with the bin file on the PC, see image

This is KModel created with a simple command as below, no input-mean or input-std used

~/nncase/ncc compile mymodel.tflite mymodel.kmodel -i tflite --dataset ~/nncase/images/

This same model when executed on K210 is producing only 7 detections.

So the INT8 quantization is not an issue.

I am currently in the same boat as you are, having different results between PC and K210.

I will recheck my code on K210, but I am confused as my current code on the K210 is still producing results with an earlier model, but doesn't produce results with this model.

[krishnak]

@sunnycase Can you help

I have compiled a kmodel, the model when tested using ncc infer is producing bounding boxes as good as the tensorflow lite model. I am using the same image to test on the PC and K210

However the kmodel doesn't produce results on K210.

So I compared the array output between K210 and that of ncc infer, following is the comparison of first 20 elements of the output array. It appears K210 output values are jumbled compared to ncc infer array - can you let me know whether this is some bug or user ignorance?

KPU | NCC infer on PC
-- | --
-0.461285 | -0.46128526
-0.461285 | -0.46128526
1E-06 | 0.46128526
1E-06 | -0.9225705
-0.92257 | -0.46128526
0.461286 | 0.46128526
-0.92257 | -0.9225705
1E-06 | 0
0.922571 | 0.46128526
-2.306426 | -2.7677116
-0.92257 | -0.46128526
1E-06 | 0
1E-06 | 0
-0.461285 | -0.9225705
-0.461285 | 0
0.461286 | 0
1E-06 | 0.46128526
-0.461285 | 0.46128526
0.461286 | -0.46128526
-0.461285 | 0.46128526

[krishnak]

I will open a new issue to clearly list out the problem, as I have isolated the reason for this disparity but I have no solution.