[Bug] Benchmark App throws Runtime Error for calibrated model

anishagartia commented 3 years ago

System information (version)

OpenVINO => 2020.4
Operating System / Platform => Windows 64 Bit
Compiler =>
Problem classification: Benchmark app on calibrated model
Framework: ONNX
Model name: 3DUNet
OpenVINO=> :grey_question:
Operating System / Platform => :grey_question:
Compiler => :grey_question:
Problem classification => :grey_question:

Detailed description

Hi, I'm running Benchmark app on a model that has been calibrated from FP32 to INT8. Uncalibrated FP32 model with same dataset run normally, I am getting this error: [Step 1/11] Parsing and validating input arguments [Step 2/11] Loading Inference Engine : : [Step 7/11] Loading the model to the device [ ERROR ] [ UNEXPECTED ] Traceback (most recent call last): File "/opt/intel/openvino_2021/python/python3.8/openvino/tools/benchmark/main.py", line 255, in run exe_network = benchmark.load_network(ie_network) File "/opt/intel/openvino_2021/python/python3.8/openvino/tools/benchmark/benchmark.py", line 60, in load_network exe_network = self.ie.load_network(ie_network, File "ie_api.pyx", line 367, in openvino.inference_engine.ie_api.IECore.load_network File "ie_api.pyx", line 379, in openvino.inference_engine.ie_api.IECore.load_network RuntimeError: [ UNEXPECTED ]

Steps to reproduce

MLPerf v1.1 Inference Datacentre offline 3D Unet https://github.com/mlcommons/inference_results_v1.1/tree/main/closed/Intel/calibration/3d-unet-99.9/openvino-cpu

Issue submission checklist

[ ] I report the issue, it's not a question
[ ] I checked the problem with documentation, FAQ, open issues, Stack Overflow, etc and have not found solution
[ ] There is reproducer code and related data files: images, videos, models, etc.

Iffa-Intel commented 3 years ago

Hi,

we are looking into this issue and will get back as soon as possible

Iffa-Intel commented 3 years ago

In order to perform Post Training Optimization with OpenVINO (Quantize FP32 into INT8), you are required to use the correct config file for the model that is going to be quantized.

May I know which .yml file you had used for this purpose? If possible could you share it? since I couldn't find specific .yml config for 3DUnet in OpenVINO (unet-camvid-onnx-0001.yml is available but not sure whether it's the right config for yours). Plus, the link you gave didn't provide the config file either.

Iffa-Intel commented 3 years ago

Closing issue, feel free to re-open or start a new issue if additional assistance is needed.

openvinotoolkit / openvino