64 bit master branch build loads model with wrong(?) parameters

[marcjasner]

I have created a build of OpenVINO from the master branch based on 64bit Ubuntu 20.04 that I am using for an Armbian 22.02 based system (Libre Computers Le Potato due to Raspberry Pi being harder to get at the moment). I used the RPi64 docker file and modified it to use Python 3.8.10 (as that's what's used in Armbian. Everything builds without errors.

I was using some of the open_model_zoo demos as a quick test to make sure it works ok, and I ran into the following problem: When using the human_pose_estimation_demo program it loads everything successfully but does not get any pose detection results. After some debugging I traced it to an issue loading the model. When loading the human-pose-estimation-0001.xml I get the following output:

[ INFO ] OpenVINO Runtime [ INFO ] build: custom_master_54e5af95da0c4288c4c9c18410748c02715f8009 [ INFO ] Reading model /home/marc/models/omz/FP16/human-pose-estimation-0001.xml [ WARNING ] The parameter "padding_mode" not found in OpenPose wrapper, will be omitted [ WARNING ] The parameter "delta" not found in OpenPose wrapper, will be omitted [ DEBUG ] Reshape model from [1, 3, 256, 456] to [1, 3, 256, 344] [ INFO ] Input layer: data, shape: [1, 3, 256, 344], precision: f32, layout: NCHW [ INFO ] Output layer: pafs, shape: [1, 38, 32, 43], precision: f32, layout: [ INFO ] Output layer: heatmaps, shape: [1, 19, 32, 43], precision: f32, layout: [ INFO ] Output layer: pooled_heatmaps, shape: [1, 19, 32, 57], precision: f32, layout:

When tracing through the code I found all results fail because at one point the code compares heatmaps with pooled_heatmaps and that check fails because they are different shaped arrays ( [1,19,32,43] vs [1,19,32,57] ).

This code works fine on my Raspberry Pi build of OpenVINO which was built from the same branch, just using the RPi32 docker file and the version of Python that Raspbian Buster uses (3.7.9). The output from the loading of the same model on the Pi shows:

[ INFO ] OpenVINO Runtime [ INFO ] build: custom_master_531fa9018d52277a4c91f3690b63382e1fc1eb94 [ INFO ] Reading model /home/pi/models/omz/FP16/human-pose-estimation-0001.xml [ WARNING ] The parameter "padding_mode" not found in OpenPose wrapper, will be omitted [ WARNING ] The parameter "delta" not found in OpenPose wrapper, will be omitted [ DEBUG ] Reshape model from [1, 3, 256, 456] to [1, 3, 256, 344] [ INFO ] Input layer: data, shape: [1, 3, 256, 344], precision: f32, layout: NCHW [ INFO ] Output layer: pafs, shape: [1, 38, 32, 43], precision: f32, layout: [ INFO ] Output layer: heatmaps, shape: [1, 19, 32, 43], precision: f32, layout: [ INFO ] Output layer: pooled_heatmaps, shape: [1, 19, 32, 43], precision: f32, layout:

You can see the pooled_heatmaps layer loads with shape [1,19,32,43], and so the comparison to heatmaps succeeds in the code. I have compared the md5sum of the model and bin files and they are identical between the two systems, so I don't see why the model loads with one shape on the 64 bit build and another shape on the 32 bit build.

Can you please assist or point me in the direction of anything that might help me get to the bottom of this?

Thanks in advance!

[jgespino]

Hi @marcjasner

Could you provide the cmake build command and steps to reproduce? I have an RPi 4 that I can test with Rasbpian OS bullseye (64 bit) and Rasbpian OS Buster (32 bit). Are you running the demo code on a MYRIAD device or on the ARM CPU?

Regards, Jesus

[marcjasner]

Here are the dockerfile I used and the build script for the docker container Armbian64_Openvino_build.zip

Additionally, if you wanted to try the binaries that my build produced, they can be found at http://www.exdomain.org/~marc/OpenVINO/OV_ARM_armbian64_package.tar.gz

I am using a MYRIAD device, but the error occurs when using CPU as well.

[marcjasner]

Hi Jesus,

Just wonder if there been any news on this issue? Have you been able to reproduce the issue? Do you need anything further from me?

Thanks!

[jgespino]

@marcjasner Apologies for the delay, not yet. The RPi4 keeps getting a kernel panic when setting up with armbian OS. I will need to build OpenVINO from source using Raspbian 64-bit (Bullseye).

[marcjasner]

No problem, just checking in. Is there anything I can do to help? Do you want me to also try building for RPi64 Bullseye? Do you have the dockerfile for that?

---

From: Jesus Espinoza @.> Sent: Tuesday, May 3, 2022 3:23 PM To: openvinotoolkit/openvino @.> Cc: Marc Jasner @.>; Mention @.> Subject: Re: [openvinotoolkit/openvino] 64 bit master branch build loads model with wrong(?) parameters (Issue #11584)

@marcjasnerhttps://github.com/marcjasner Apologies for the delay, not yet. The RPi4 keeps getting a kernel panic when setting up with armbian OS. I will need to build OpenVINO from source using Raspbian 64-bit (Bullseye).

— Reply to this email directly, view it on GitHubhttps://github.com/openvinotoolkit/openvino/issues/11584#issuecomment-1116475492, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX7CRNW5GOWWBLX2KCXQ2ZTVIF4KFANCNFSM5UMEPVNQ. You are receiving this because you were mentioned.Message ID: @.***>

[jgespino]

@marcjasner if you could try it on bullseye that would be great! We have a guide for Raspbian OS Stretch but it should work for Bullseye too. https://www.intel.com/content/www/us/en/support/articles/000057005/boards-and-kits.html

Just clone the 2022.1.0 repos: git clone https://github.com/openvinotoolkit/openvino -b 2022.1.0 git clone https://github.com/opencv/opencv -b 4.5.5-openvino-2022.1.0

[marcjasner]

I will try it and let you know what happens. Thanks, Marc

---

From: Jesus Espinoza @.> Sent: Tuesday, May 3, 2022 4:06 PM To: openvinotoolkit/openvino @.> Cc: Marc Jasner @.>; Mention @.> Subject: Re: [openvinotoolkit/openvino] 64 bit master branch build loads model with wrong(?) parameters (Issue #11584)

@marcjasnerhttps://github.com/marcjasner if you could try it on bullseye that would be great! We have a guide for Raspbian OS Stretch but it should work for Bullseye too. https://www.intel.com/content/www/us/en/support/articles/000057005/boards-and-kits.html

Just clone the 2022.1.0 repos: git clone https://github.com/openvinotoolkit/openvino -b 2022.1.0 git clone https://github.com/opencv/opencv -b 4.5.5-openvino-2022.1.0

— Reply to this email directly, view it on GitHubhttps://github.com/openvinotoolkit/openvino/issues/11584#issuecomment-1116532232, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX7CRNXBFMDPLJ4QOQYJG2DVIGBNLANCNFSM5UMEPVNQ. You are receiving this because you were mentioned.Message ID: @.***>

[marcjasner]

Is this to build it on the device directly?

---

From: Marc Jasner @.> Sent: Tuesday, May 3, 2022 4:08 PM To: openvinotoolkit/openvino @.>; openvinotoolkit/openvino @.> Cc: Mention @.> Subject: Re: [openvinotoolkit/openvino] 64 bit master branch build loads model with wrong(?) parameters (Issue #11584)

I will try it and let you know what happens. Thanks, Marc

---

From: Jesus Espinoza @.> Sent: Tuesday, May 3, 2022 4:06 PM To: openvinotoolkit/openvino @.> Cc: Marc Jasner @.>; Mention @.> Subject: Re: [openvinotoolkit/openvino] 64 bit master branch build loads model with wrong(?) parameters (Issue #11584)

@marcjasnerhttps://github.com/marcjasner if you could try it on bullseye that would be great! We have a guide for Raspbian OS Stretch but it should work for Bullseye too. https://www.intel.com/content/www/us/en/support/articles/000057005/boards-and-kits.html

Just clone the 2022.1.0 repos: git clone https://github.com/openvinotoolkit/openvino -b 2022.1.0 git clone https://github.com/opencv/opencv -b 4.5.5-openvino-2022.1.0

— Reply to this email directly, view it on GitHubhttps://github.com/openvinotoolkit/openvino/issues/11584#issuecomment-1116532232, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX7CRNXBFMDPLJ4QOQYJG2DVIGBNLANCNFSM5UMEPVNQ. You are receiving this because you were mentioned.Message ID: @.***>

[jgespino]

@marcjasner Yes, it will build OpenVINO for runtime only. You will need another system with the OpenVINO dev tools to compile the model and copy it over the IR files.

[marcjasner]

Success! It took a long time to build but that seems to be working correctly now!

Now I just need to figure out how to zip it all up like I the docker build does so I can reproduce this setup without having to rebuild from scratch every time.

Thanks again for the help. Shall I close the bug?

[jgespino]

That was fast! Mine is still building... Feel free to close the bug and start a new issue if you have any additional questions.

[marcjasner]

I don't know that I'd call it fast, but it definitely built and ran successfully! I did have to add a fair amount of swap to the armbian build to keep it building, but whatever works!

I'll close this. Thanks

---

From: Jesus Espinoza @.> Sent: Thursday, May 5, 2022 3:06 PM To: openvinotoolkit/openvino @.> Cc: Marc Jasner @.>; Mention @.> Subject: Re: [openvinotoolkit/openvino] 64 bit master branch build loads model with wrong(?) parameters (Issue #11584)

That was fast! Mine is still building... Feel free to close the bug and start a new issue if you have any additional questions.

— Reply to this email directly, view it on GitHubhttps://github.com/openvinotoolkit/openvino/issues/11584#issuecomment-1118952339, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AX7CRNSGPMF6ZPBFO3UYTADVIQL5BANCNFSM5UMEPVNQ. You are receiving this because you were mentioned.Message ID: @.***>