[Bug]: Model infers on Intel CPU, but crashes on ARM CPU (both systems using Ubuntu 22.04)

kmn1024 commented 2 months ago

OpenVINO Version

2024.1.0

Operating System

Other (Please specify in description)

Device used for inference

CPU

Framework

None

Model used

Custom (a version of Hifi-GAN)

Issue description

This model is a version of Hifi-GAN with some customizations. I converted it from Pytorch on my desktop, which runs an Intel CPU on Ubuntu 22.04, using openvino-2024.1.0-15008-cp310-cp310-manylinux2014_x86_64.whl, using these instructions: https://github.com/openvinotoolkit/openvino/blob/74829b1ad22fdc5cd915bd0ec1bba5a4c20cfe08/docs/articles_en/openvino-workflow/model-preparation.rst#convert-a-model-with-python-convert_model

On the desktop, the model loads (ov.compile_model) and infers perfectly fine.

However, if I move the model to an ARM-based edge computer (Orange Pi 5, which has A76+A55 CPU), with openvino-2024.1.0-15008-cp312-cp312-manylinux_2_31_aarch64.whl.metadata installed, the model loads but inference crashes. Stack trace of the crash:

(gdb) bt
#0  0x0000007f3754aa14 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#1  0x0000007f3689f2cc in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#2  0x0000007f3689f828 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#3  0x0000007f6cc66d88 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/../openvino/libs/libtbb.so.12
#4  0x0000007f6cc63290 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/../openvino/libs/libtbb.so.12
#5  0x0000007f368a0068 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#6  0x0000007f368a5574 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#7  0x0000007f374fb584 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#8  0x0000007f368a09ac in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#9  0x0000007f36974d58 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#10 0x0000007f36960910 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#11 0x0000007f3677adb8 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#12 0x0000007f36730ff4 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#13 0x0000007f36740bc0 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#14 0x0000007f3674f8f0 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/libs/libopenvino_arm_cpu_plugin.so
#15 0x0000007f69c6b15c in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/../openvino/libs/libopenvino.so.2410
#16 0x0000007f69c8283c in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/../openvino/libs/libopenvino.so.2410
#17 0x0000007f6cc4a8c4 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/../openvino/libs/libtbb.so.12
#18 0x0000007f69c87b00 in ?? () from /home/orangepi/miniconda3/envs/platform_openvino/lib/python3.12/site-packages/openvino/../openvino/libs/libopenvino.so.2410
#19 0x0000007f821431fc in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#20 0x0000007f8496d5c8 in ?? () from /lib/aarch64-linux-gnu/libc.so.6
#21 0x0000007f849d5d9c in ?? () from /lib/aarch64-linux-gnu/libc.so.6

I have attached the xml portion of the saved model: decoder2-openvino-xml.txt

Step-by-step reproduction

Difficult. The .bin portion of the saved model is about 90MB, so I cannot upload it. Please let me know if this is absolutely required.

Relevant log output

No response

Issue submission checklist

[X] I'm reporting an issue. It's not a question.
[X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
[X] There is reproducer code and related data files such as images, videos, models, etc.

allnes commented 2 months ago

@kmn1024 Hi, thank you for issue. Could you provide yours custom pytorch model or model in OpenVINO IR after conversion (.xml and .bin files)? Upd. if it posiible please provide link on the storage with model

kmn1024 commented 2 months ago

Thanks for looking, @allnes !

The .bin file: https://mega.nz/file/FalykSAS#IgHmpV_LGO56U1Cdeh2ko9Ggkj7hp9uiw9oyQI9ZAtM

The .xml is attached in the original post (as decoder2-openvino-xml.txt)

allnes commented 2 months ago

Thanks for looking, @allnes !

The .bin file: https://mega.nz/file/FalykSAS#IgHmpV_LGO56U1Cdeh2ko9Ggkj7hp9uiw9oyQI9ZAtM

The .xml is attached in the original post (as decoder2-openvino-xml.txt)

Thanks for model. I will return when I get some results.

allnes commented 1 month ago

@kmn1024 hi! Could you provide yours script of the model conversion? Because yours model (.xml and .bin) has an internal defect.

kmn1024 commented 1 month ago

# Prepare pytorch model
...
decoder_model.eval()

import openvino as ov
ov_model = ov.convert_model(decoder_model,
                            input={
                                'd': ov.PartialShape([1, -1, 640]),
                                't_en': ov.PartialShape([1, 512, -1]),
                                'pred_aln_trg': ov.PartialShape([-1, -1]),
                                's': ov.Shape([1, 128]),
                                'ref': ov.Shape([1, 128]),
                            },
                            example_input=(d, t_en, pred_aln_trg, s, ref))

kmn1024 commented 1 month ago

@allnes Can you please let me know if the above is what you need to help you debug, or do you need something else?

I can also do a bit of debugging on my end, if you can guide me on where to look.

allnes commented 1 month ago

@kmn1024 Alas, I could not reproduce this case, then I would like to ask you to build an OpenVINO library with DEBUG level and send us a stack trace of the network inference crash.

openvinotoolkit / openvino