search

openvinotoolkit / openvino

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference
https://docs.openvino.ai
Apache License 2.0
7.25k stars 2.26k forks source link

[Bug]: Port for tensor name x was not found #21615

Open jikechao opened 11 months ago

jikechao commented 11 months ago

OpenVINO Version

openvino-nightly 2023.2.0.dev20231101

Operating System

Ubuntu 18.04 (LTS)

Device used for inference

CPU

Framework

ONNX

Model used

https://github.com/jikechao/onnx_models/blob/main/dropout.onnx

Issue description

For the given model containing Dropout operator, OpenVINO can convert it into OV IR correctly and compile it well. However, when executing inference, it will report an error: Port for tensor name x was not found. I consider such behavior a bug because OpenVINO doesn't align with the ONNXRuntime rather than crash.

Step-by-step reproduction

import openvino as ov
import numpy as np

ov_model = ov.convert_model('dropout.onnx')  # input=input_shapes

ir_path = f"temp_OVIR.xml"
ov.save_model(ov_model, ir_path, compress_to_fp16=False)
core = ov.Core()
model = core.read_model(ir_path)

compiled_model = core.compile_model(model=model, device_name="CPU")

input_x = np.random.random([3,  4, 5]).astype(np.float32)
input_data = {"x": input_x}

result = []
for output in ov_model.outputs:
    result.append(compiled_model(input_data)[output])

Relevant log output

Traceback (most recent call last):
  File "test.py", line 20, in <module>
    result.append(compiled_model(input_data)[output])
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\ie_api.py", line 384, in __call__
    return self._infer_request.infer(
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\ie_api.py", line 143, in infer
    return OVDict(super().infer(_data_dispatch(
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 354, in _data_dispatch
    return create_shared(inputs, request) if is_shared else create_copied(inputs, request)
  File "C:\software\conda\envs\torch\lib\functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 182, in _
    return {k: value_to_tensor(v, request=request, is_shared=True, key=k) for k, v in request._inputs_data.items()}
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 182, in <dictcomp>
    return {k: value_to_tensor(v, request=request, is_shared=True, key=k) for k, v in request._inputs_data.items()}
  File "C:\software\conda\envs\torch\lib\functools.py", line 877, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 59, in _
    tensor = get_request_tensor(request, key)
  File "C:\software\conda\envs\torch\lib\site-packages\openvino\runtime\utils\data_helpers\data_dispatcher.py", line 27, in get_request_tensor
    return request.get_tensor(key)
RuntimeError: Exception from src\inference\src\infer_request.cpp:194:
Check '::getPort(port, name, {_impl->get_inputs(), _impl->get_outputs()})' failed at src\inference\src\infer_request.cpp:194:
Port for tensor name x was not found.

Issue submission checklist

avitial commented 10 months ago

@jikechao the provided ONNX model executes fine while using the benchmark_app on OpenVINO 2023.2 (see below), please double check the code snippet you've provided. Check the Inference Pipeline documentation for reference. 

$ benchmark_app -m dropout.onnx -d CPU -niter 1000
[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2023.2.0-13089-cfd42bd2cb0-HEAD
[ INFO ]
[ INFO ]
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 7.45 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     y (node: y) : f32 / [...] / [3,4,5]
[ INFO ] Model outputs:
[ INFO ]     y (node: y) : f32 / [...] / [3,4,5]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     y (node: y) : u8 / [C,H,W] / [3,4,5]
[ INFO ] Model outputs:
[ INFO ]     ***NO_NAME*** (node: Convert_4) : f32 / [...] / [3,4,5]
[Step 7/11] Loading the model to the device
[ INFO ] Compile model took 7.32 ms
[Step 8/11] Querying optimal runtime parameters
[ INFO ] Model:
[ INFO ]   NETWORK_NAME: Dropout
[ INFO ]   OPTIMAL_NUMBER_OF_INFER_REQUESTS: 4
[ INFO ]   NUM_STREAMS: 4
[ INFO ]   AFFINITY: Affinity.CORE
[ INFO ]   INFERENCE_NUM_THREADS: 8
[ INFO ]   PERF_COUNT: False
[ INFO ]   INFERENCE_PRECISION_HINT: <Type: 'float32'>
[ INFO ]   PERFORMANCE_HINT: PerformanceMode.THROUGHPUT
[ INFO ]   EXECUTION_MODE_HINT: ExecutionMode.PERFORMANCE
[ INFO ]   PERFORMANCE_HINT_NUM_REQUESTS: 0
[ INFO ]   ENABLE_CPU_PINNING: True
[ INFO ]   SCHEDULING_CORE_TYPE: SchedulingCoreType.ANY_CORE
[ INFO ]   ENABLE_HYPER_THREADING: True
[ INFO ]   EXECUTION_DEVICES: ['CPU']
[ INFO ]   CPU_DENORMALS_OPTIMIZATION: False
[ INFO ]   CPU_SPARSE_WEIGHTS_DECOMPRESSION_RATE: 1.0
[Step 9/11] Creating infer requests and preparing input tensors
[ WARNING ] No input files were given for input 'y'!. This input will be filled with random values!
[ INFO ] Fill input 'y' with random values
[Step 10/11] Measuring performance (Start inference asynchronously, 4 inference requests, limits: 1000 iterations)
[ INFO ] Benchmarking in inference only mode (inputs filling are not included in measurement loop).
[ INFO ] First inference took 0.13 ms
[Step 11/11] Dumping statistics report
[ INFO ] Execution Devices:['CPU']
[ INFO ] Count:            1000 iterations
[ INFO ] Duration:         19.88 ms
[ INFO ] Latency:
[ INFO ]    Median:        0.02 ms
[ INFO ]    Average:       0.03 ms
[ INFO ]    Min:           0.01 ms
[ INFO ]    Max:           10.24 ms
[ INFO ] Throughput:   50311.93 FPS
jikechao commented 10 months ago

Hi @avitial, I updated the OpenVINO version to 2024.0.0-13874-d7ba44f8c38, Crash still occurred。 BTW, what does the command benchmark_app refer to?

avitial commented 10 months ago

@jikechao the benchmark_app refers to the Benchmark Python Tool used to estimate deep learning inference performance on supported devices. This tool is automatically installed when you install OpenVINO Developer Tools using PyPI, check the docs for more info.

andrei-kochin commented 6 months ago

@jikechao I see that the issue lies in translation where Dropout layer just got disappeared and model just contains result node and x input indeed missing.

IR file:

<?xml version="1.0"?>
<net name="Dropout" version="11">
    <layers>
        <layer id="0" name="y" type="Parameter" version="opset1">
            <data shape="3,4,5" element_type="f32" />
            <output>
                <port id="0" precision="FP32" names="y">
                    <dim>3</dim>
                    <dim>4</dim>
                    <dim>5</dim>
                </port>
            </output>
        </layer>
        <layer id="1" name="y/sink_port_0" type="Result" version="opset1">
            <input>
                <port id="0" precision="FP32">
                    <dim>3</dim>
                    <dim>4</dim>
                    <dim>5</dim>
                </port>
            </input>
        </layer>
    </layers>
    <edges>
        <edge from-layer="0" from-port="0" to-layer="1" to-port="0" />
    </edges>
    <rt_info />
</net>

ONNX infer script:

import onnxruntime as rt
import numpy as np

sess = rt.InferenceSession("dropout.onnx")
input= np.ndarray([3, 4, 5], dtype=np.single)
sess.run([], {'x': input})
andrei-kochin commented 6 months ago

Legacy MO ONNX which can translate this op to Identity and input survives:

https://github.com/openvinotoolkit/openvino/blob/494736918d1a3fef0c49eae8b8366e2832c0c00d/tools/mo/openvino/tools/mo/front/onnx/dropout_ext.py#L10

    <layers>
        <layer id="0" name="x" type="Parameter" version="opset1">
            <data shape="3,4,5" element_type="f32" />
            <output>
                <port id="0" precision="FP32" names="x,x:0">
                    <dim>3</dim>
                    <dim>4</dim>
                    <dim>5</dim>
                </port>
            </output>
        </layer>
jikechao commented 2 months ago

@avitial @andrei-kochin Thanks for your deep investigation and the workaround to bypass this bug.

BTW, it seems that Legacy MO has been deprecated recently. image

andrei-kochin commented 2 months ago

@jikechao indeed. My last message was more as a reference for developers not as a solution. However if it is urgent for you you can try the legacy ONNX to generate the IR. Legacy behavior can be used as a reference code to align behavior in ONNX FE.

jikechao commented 2 months ago

@andrei-kochin Got it. Thank you!