what's the differences between onnxruntime with openvino backend VS openvino directly?

[lucasjinreal]

Describe the issue

what's the differences between onnxruntime with openvino backend VS openvino directly?

To reproduce

what's the differences between onnxruntime with openvino backend VS openvino directly?

Urgency

No response

Platform

Linux

OS Version

22.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

436576

ONNX Runtime API

C++

Architecture

X86

Execution Provider

Default CPU, CoreML

Execution Provider Library Version

No response

[jywu-msft]

OnnxRuntime can delegate portions of the ONNX graph (subgraphs) to execution providers that are enabled. OpenVINO does not support all ONNX operations. So OnnxRuntime + OpenVINO EP is able to support more ONNX models than OpenVINO on its own. This is true for other execution providers as well. see my response to https://github.com/microsoft/onnxruntime/issues/12083

[lucasjinreal]

@jywu-msft thanks. From my understanding, onnxruntime will using vendor API to execution certain op, but my question is, for different Vendor, their API mgiht not same as onnxrutnime itself, in this situation, how will it convert? For example, NMS should be different among all frameworks.

On the other hand, the graph optimization in OpenVINO might didn't in onnxruntime, they can do their own passes, to suiting their simplified op (for less mem access), so that some ops actually didn't really fit onnx standared. How does onnxruntime compelete this compare with pure openvino?

[jywu-msft]

@jywu-msft thanks. From my understanding, onnxruntime will using vendor API to execution certain op, but my question is, for different Vendor, their API mgiht not same as onnxrutnime itself, in this situation, how will it convert? For example, NMS should be different among all frameworks.

On the other hand, the graph optimization in OpenVINO might didn't in onnxruntime, they can do their own passes, to suiting their simplified op (for less mem access), so that some ops actually didn't really fit onnx standared. How does onnxruntime compelete this compare with pure openvino?

if i'm understanding your question correctly, you're asking if the kernel implementation in the native framework isn't compatible with ONNX or isn't optimized for ONNX, what happens? if they don't support the ONNX spec of the op, then there's not much we can do, since we are targeting ONNX models.