Inference on Edge TPU

[Corbaci57]

Hi, thanks for the project.

I am still new to AI and would like to run the project with a RPI4 in combination with coral usb accelerator (with NCS2 works fine).

For this I have converted the model. The input shape was changed to 1x256x448x3. This corresponds to the original format of the scaled_img. In addition, I have adapted the InferenceEngine accordingly.

However, I get different results after executing the code, which later causes a memory access error.

I hope you have an idea where my error is and how I can make it right.

Part of the results look like this (maybe it is helpful) :

features values form tflite: [[[[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 3.4559049e-02 -6.9118097e-02 -5.5294476e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 4.8382670e-02 -5.5294476e-02 -4.8382666e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 3.4559049e-02 -3.4559049e-02 -6.2206283e-02] ... [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -1.3823618e-02 1.9353066e-01 -4.1470855e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -1.1058895e-01 8.2941718e-02 1.3823620e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -2.7647238e-02 -6.9118097e-02 2.0735430e-02]]

[[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 2.7647238e-02 -8.9853525e-02 -4.8382666e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 2.0735430e-02 2.0735430e-02 -4.1470855e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -3.4559049e-02 3.4559049e-02 -4.8382666e-02] ... [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -4.8382666e-02 2.7647239e-01 -6.9118086e-03] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -1.0367714e-01 1.1058895e-01 4.1470859e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -4.1470855e-02 -6.9118097e-02 2.0735430e-02]]

[[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 2.7647238e-02 -8.2941711e-02 -4.8382666e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -2.7647238e-02 3.4559049e-02 -2.0735428e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -6.9118097e-02 9.6765332e-02 -3.4559049e-02] ... [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 3.1103143e-01 2.0735430e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -5.5294476e-02 1.3823619e-01 5.5294476e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -4.1470855e-02 -5.5294476e-02 1.3823620e-02]]

...

[[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 4.8382670e-02 -4.8382666e-02 2.0735430e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -1.3823618e-02 4.1470859e-02 1.7279524e-01] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -2.7647238e-02 6.9118097e-02 2.2117791e-01] ... [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 9.3132257e-10 -6.2206283e-02 -1.3823618e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 1.3823620e-02 -6.2206283e-02 -2.0735428e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -6.9118086e-03 -6.9118097e-02 -6.9118086e-03]]

[[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 4.8382670e-02 -4.8382666e-02 1.3823620e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 4.8382670e-02 1.5205981e-01] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 4.8382670e-02 2.1426609e-01] ... [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 -6.9118097e-02 -1.3823618e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 -6.2206283e-02 -2.0735428e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 -7.6029904e-02 -6.9118086e-03]]

[[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 3.4559049e-02 -4.1470855e-02 4.8382670e-02] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... -1.3823618e-02 6.2206287e-02 1.7279524e-01] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 9.3132257e-10 4.1470859e-02 2.1426609e-01] ... [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 3.4559049e-02 -7.6029904e-02 -6.9118086e-03] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 2.7647238e-02 -7.6029904e-02 -6.9118086e-03] [ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 6.9118105e-03 -8.2941711e-02 6.9118105e-03]]]]

features values from openvino_model: [[[-2.44140625e-04 -2.13623047e-04 -2.05993652e-04 ... -3.28063965e-04 -2.97546387e-04 -2.28881836e-04] [-2.21252441e-04 -2.05993652e-04 -1.75476074e-04 ... -2.59399414e-04 -2.44140625e-04 -2.59399414e-04] [-2.21252441e-04 -1.44958496e-04 -1.14440918e-04 ... -2.59399414e-04 -2.36511230e-04 -2.36511230e-04] ... [-1.98364258e-04 -2.13623047e-04 -1.75476074e-04 ... -1.83105469e-04 -1.83105469e-04 -1.75476074e-04] [-2.05993652e-04 -2.44140625e-04 -1.98364258e-04 ... -1.83105469e-04 -1.90734863e-04 -2.21252441e-04] [-2.44140625e-04 -1.98364258e-04 -2.82287598e-04 ... -1.52587891e-04 -1.90734863e-04 -1.52587891e-04]]

[[-3.28063965e-04 -9.91821289e-05 -1.60217285e-04 ... -1.44958496e-04 -2.51770020e-04 -2.21252441e-04] [-1.83105469e-04 0.00000000e+00 -1.37329102e-04 ... -1.52587891e-04 -2.44140625e-04 -2.51770020e-04] [-2.05993652e-04 0.00000000e+00 0.00000000e+00 ... -3.66210938e-04 -2.97546387e-04 -1.60217285e-04] ... [-1.90734863e-04 -6.10351562e-05 1.83105469e-04 ... 6.86645508e-05 0.00000000e+00 -1.06811523e-04] [-1.83105469e-04 0.00000000e+00 1.14440918e-04 ... 0.00000000e+00 0.00000000e+00 0.00000000e+00] [-2.82287598e-04 0.00000000e+00 -1.29699707e-04 ... -1.14440918e-04 0.00000000e+00 -9.91821289e-05]]

[[-4.65393066e-04 -4.11987305e-04 -3.73840332e-04 ... -3.73840332e-04 -4.19616699e-04 -4.65393066e-04] [-4.50134277e-04 -4.80651855e-04 -3.96728516e-04 ... -4.80651855e-04 -4.95910645e-04 -4.50134277e-04] [-4.57763672e-04 -4.73022461e-04 -4.73022461e-04 ... -5.11169434e-04 -4.11987305e-04 -4.42504883e-04] ... [-4.50134277e-04 -4.73022461e-04 -4.27246094e-04 ... -3.50952148e-04 -3.58581543e-04 -3.89099121e-04] [-4.42504883e-04 -4.34875488e-04 -4.27246094e-04 ... -3.89099121e-04 -3.66210938e-04 -3.58581543e-04] [-4.57763672e-04 -4.65393066e-04 -5.03540039e-04 ... -3.96728516e-04 -3.66210938e-04 -4.04357910e-04]]

...

[[ 3.30810547e-02 3.55529785e-02 2.82897949e-02 ... -5.01403809e-02 -1.05651855e-01 -4.50134277e-02] [ 2.61383057e-02 2.57720947e-02 2.22473145e-02 ... -1.08093262e-01 -1.07727051e-01 -7.55004883e-02] [ 3.06091309e-02 4.54101562e-02 3.17382812e-02 ... -1.33422852e-01 -1.08215332e-01 -6.64672852e-02] ... [ 2.83813477e-02 1.57470703e-02 2.27050781e-02 ... -7.24487305e-02 -3.86657715e-02 -5.70373535e-02] [ 3.02276611e-02 3.09448242e-02 4.38232422e-02 ... -3.70788574e-02 -1.86462402e-02 -3.84826660e-02] [ 3.55224609e-02 -1.41296387e-02 2.09655762e-02 ... -2.11486816e-02 -2.07214355e-02 -4.09240723e-02]]

[[-7.28149414e-02 -8.08715820e-02 -9.50317383e-02 ... 9.97314453e-02 4.61120605e-02 -4.08630371e-02] [-8.20312500e-02 -7.01293945e-02 -8.61206055e-02 ... 1.18774414e-01 6.27441406e-02 -2.68096924e-02] [-7.11059570e-02 -7.59887695e-02 -8.07495117e-02 ... 1.32324219e-01 3.90319824e-02 -2.95562744e-02] ... [-6.37817383e-02 -3.49426270e-02 2.21252441e-03 ... -4.04663086e-02 -5.35278320e-02 -5.59997559e-02] [-6.97021484e-02 -4.21752930e-02 -1.54113770e-03 ... -5.98449707e-02 -7.10449219e-02 -7.20214844e-02] [-5.45654297e-02 3.93371582e-02 7.75756836e-02 ... -6.37817383e-02 -6.73828125e-02 -7.04345703e-02]]

[[-3.30810547e-02 -1.82495117e-02 -1.69372559e-02 ... -1.94244385e-02 -6.71386719e-03 2.88238525e-02] [-2.57568359e-02 -1.54495239e-02 -2.28881836e-04 ... -4.48608398e-03 5.11169434e-03 2.99224854e-02] [-2.84271240e-02 -9.94873047e-03 -1.35192871e-02 ... 1.59149170e-02 2.22015381e-02 1.96380615e-02] ... [-2.91137695e-02 1.09405518e-02 3.97338867e-02 ... 6.98242188e-02 2.69927979e-02 2.35137939e-02] [-3.27758789e-02 1.03759766e-03 3.13110352e-02 ... 3.80859375e-02 4.89807129e-03 1.16729736e-02] [ 4.80651855e-03 9.50927734e-02 1.45996094e-01 ... 1.78070068e-02 2.21405029e-02 1.81427002e-02]]]

[Daniil-Osokin]

Hi! I suggest to check Inference with OpenVINO section, does it work on CPU?

[Corbaci57]

Thanks for your response.

Yes it works.

I managed to get the program to start even with the TFlite model. Input Shape and Output Shape had to be adjusted because they were changed during the converting process.

However, nothing is recognized since the results of the models are still very different.

For example the TFlite returns values for features like

[ 9.3132257e-10 9.3132257e-10 9.3132257e-10 ... 3.4559049e-02 -6.9118097e-02 -5.5294476e-02]

Although the model for OpenVINO returns values for features like

[-2.44140625e-04 -2.13623047e-04 -2.05993652e-04 ... -3.28063965e-04 -2.97546387e-04 -2.28881836e-04]

[Daniil-Osokin]

If the original demo with OpenVINO works on a CPU, then just pass --use-openvino --device MYRIAD to the demo script to run it on NCS2. As I get you are trying to convert the PyTorch model to OpenVINO format and run the demo with this converted model. I just did not get why the channels dimension is the last in the shape you provided, it must be on the second place for the OpenVINO: 1x256x448x3 -> 1x3x256x448.

[Corbaci57]

I want to run it on a Raspberry with a Coral USB accelerator. For this the ONNX model was converted to a TFlite model. Why the converter changes the shape, I do not know unfortunately.

[Daniil-Osokin]

So, you want to run the demo through TFLite on Coral accelerator, not OpenVINO with NCS2, and adapted the code somehow. Did you feed normalized_image as is (normalized_image[np.newaxis,]), since TF expects channels dimension to be the last?

[Daniil-Osokin]

Also what is the shape of output? If it is 1x32x56xNUM_CHANNELS, it need to be transposed to also has channels on the second place, not the 4th to work with existing code.

[Corbaci57]

I used the inference_engine_openvino.py as a template, there was no normalization there.

I'll try normalization. Thanks.

Yes, the output shape is also different. I transposed it.

[Corbaci57]

Thanks, it works with normalization!

Could you briefly explain to me why you don't need normalization with OpenVINO, but here it is?

I also get the following message: "Cannot load fast pose extraction, switched to legacy slow implementation."

How can I load the fast pose extraction?

[Daniil-Osokin]

Normalization already included in the OpenVINO model. To load fast pose extractor, please follow prerequisites section.

[Corbaci57]

Thanks!