search

microsoft / onnxruntime

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator
https://onnxruntime.ai
MIT License
14.45k stars 2.9k forks source link

onnxruntime openvino docker inference very slow #4029

Closed happog closed 4 years ago

happog commented 4 years ago

I pull onnxruntime openvino docker using: docker pull mcr.microsoft.com/azureml/onnxruntime:latest-openvino-cpu

When I run a model in the docker, the inference is very slow (45 seconds), and output the following info:

[WARN] 2020-05-26T06:04:53z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph [WARN] 2020-05-26T06:04:53z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph [WARN] 2020-05-26T06:04:53z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.preview.training' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.nchwc' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.ml' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'com.microsoft.mlfeaturizers' not recognized by nGraph [WARN] 2020-05-26T06:04:54z src/ngraph/frontend/onnx_import/ops_bridge.cpp 190 Domain 'ai.onnx.training' not recognized by nGraph

jywu-msft commented 4 years ago

+@smkarlap , @suryasidd to take a look. can you share the model? the first inference can be slow, can you confirm that subsequent inferences are fast compared to the first one?

suryasidd commented 4 years ago

Could you also send us the verbose logs please? If you are using either onnx_test_runner or onnxruntime_perf_test, just adding -v option to the command should do. If you are running a python sample, onnxruntime.set_default_logger_severity(0) this will give the verbose logs.

happog commented 4 years ago

Thank you, actually, the following iteration is quite fast!