[Build]: Dynamic Input Issue on NPU with GNN Inference

[Endlessfancy]

OpenVINO Version

2024.03

Operating System

Windows System

Hardware Architecture

x86 (64 bits)

Target Platform

Host Name: LAPTOP-D60VPN1Q OS Name: Microsoft Windows 11 家庭中文版 OS Version: 10.0.22631 N/A Build 22631 OS Manufacturer: Microsoft Corporation OS Configuration: Standalone Workstation OS Build Type: Multiprocessor Free Registered Owner: Haoyang Fan Registered Organization: N/A Product ID: 00342-31692-69479-AAOEM Original Install Date: 7/22/2024, 2:47:39 AM System Boot Time: 9/2/2024, 5:41:49 PM System Manufacturer: LENOVO System Model: 21LE System Type: x64-based PC Processor(s): 1 Processor(s) Installed. 01: Intel64 Family 6 Model 170 Stepping 4 GenuineIntel ~2300 Mhz BIOS Version: LENOVO NJCN44WW, 5/14/2024 Windows Directory: C:\Windows System Directory: C:\Windows\system32 Boot Device: \Device\HarddiskVolume1 System Locale: en-us;English (United States) Input Locale: en-us;English (United States) Time Zone: (UTC-08:00) Pacific Time (US & Canada) Total Physical Memory: 65,141 MB Available Physical Memory: 52,558 MB Virtual Memory: Max Size: 74,869 MB Virtual Memory: Available: 60,685 MB Virtual Memory: In Use: 14,184 MB Page File Location(s): C:\pagefile.sys Domain: WORKGROUP Logon Server: \LAPTOP-D60VPN1Q Hotfix(s): 4 Hotfix(s) Installed.

                       [02]: KB5027397
                       [03]: KB5041585
                       [04]: KB5041584

Network Card(s): 3 NIC(s) Installed. 01: Intel(R) Ethernet Connection (18) I219-V Connection Name: 以太网 Status: Media disconnected [02]: Intel(R) Wi-Fi 6E AX211 160MHz Connection Name: WLAN DHCP Enabled: Yes DHCP Server: 192.168.0.1 IP address(es)

                             [02]: fe80::ec79:5f52:b5c6:5427
                             [03]: fdc2:29b:a9d3:1:98ea:ad15:939e:1ce7
                             [04]: fd71:972b:863d:8044:98ea:ad15:939e:1ce7
                             [05]: fdc2:29b:a9d3:1:37bd:d31a:79:c15c
                             [06]: fd71:972b:863d:8044:6224:225:fe5c:5fd3
                       [03]: Bluetooth Device (Personal Area Network)
                             Connection Name: 蓝牙网络连接
                             Status:          Media disconnected

Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.

Build issue description

I encountered an issue while running GNN benchmarks for inference on OpenVINO. The GNN models I am using have varying batch sizes, The inference runs smoothly on CPU and GPU, but when running on NPU, I encountered the following error: “to_shape was called on a dynamic shape”

I noticed that the documentation and Issue mentions that NPU currently does not support dynamic inputs.

I would like to inquire whether there is any support for dynamic inputs in NPU inference or if there are alternative methods to handle varying batch sizes during inference.

Build script or step-by-step to reproduce

OS system: Windows
OpenVINO version: 2024.03
Python virtual env version: python3.10

1. Use a GNN model with varying batch sizes. Here is the code.

2. Prepare the model for ONNX export with dynamic input settings:

Dummy_input = (data.x, data.edge_index) torch.onnx.export( model, dummy_input, model_onnx_path, input_names=['node_features', 'edge_index'], output_names=['output'], dynamic_axes={'node_features': {0: 'num_nodes'},
'edge_index': {1: 'num_edges'}}, opset_version=14 )

3. Convert the ONNX model to an IR model using the Model Optimizer (mo).

4. Run inference on NPU with the following code

core = ov.Core() compiled_model = core.compile_model(model=model_ir_path, device_name="NPU")

Relevant log output

The model fails to run on NPU, throwing the error:  

to_shape was called on a dynamic shape. 

The same model with static input sizes runs without issues on NPU.

Issue submission checklist

• [X] I'm reporting an issue. It's not a question.
• [X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.

[rkazants]

@rzubarev, @mlyashko, please take a look.

Regards, Roman

[YuChern-Intel]

Currently, only the models with static shapes are supported on NPU. Dynamic shapes are not supported by the NPU plugin yet.

[avitial]

Ref. 149740