search

sony / model_optimization

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.
https://sony.github.io/model_optimization/
Apache License 2.0
297 stars 49 forks source link

Unable to load yolov8 exported with ptq through onnx #1188

Closed ambitious-octopus closed 2 weeks ago

ambitious-octopus commented 2 weeks ago

Issue Type

Bug

Source

pip (mct-nightly)

MCT Version

2.1.0

OS Platform and Distribution

Linux Ubuntu 22.04

Python version

3.10

Describe the issue

I followed the tutorial provided in tutorials/notebooks/imx500_notebooks/pytorch/pytorch_yolov8n_for_imx500.ipynb and successfully exported a model in ONNX format using MCT PTQ. However, when I attempt to load the model with ONNX, I encounter the following error:

Traceback (most recent call last):
  File "/home/me/repos/ultralytics/test.py", line 13, in <module>
    ort_sess = ort.InferenceSession('yolov8n_mct_model.onnx')
  File "/home/me/miniconda3/envs/ultralytics/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 419, in __init__
    self._create_inference_session(providers, provider_options, disabled_optimizers)
  File "/home/me/miniconda3/envs/ultralytics/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 480, in _create_inference_session
    sess = C.InferenceSession(session_options, self._model_path, True, self._read_config_from_model)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from yolov8n_mct_model.onnx failed:Fatal error: mct_quantizers:ActivationPOTQuantizer(-1) is not a registered function/op

cc: @Idan-BenAmi

Expected behaviour

No response

Code to reproduce the issue

import onnx
import onnxruntime as ort
onnx_model = onnx.load("yolov8n_mct_model.onnx")
onnx.checker.check_model(onnx_model)
ort_sess = ort.InferenceSession('yolov8n_mct_model.onnx')

Log output

No response

ambitious-octopus commented 2 weeks ago

Update: found the notebook tutorials/notebooks/mct_features_notebooks/pytorch/example_pytorch_export.ipynb with an example of onnx import. It works, now I have a new missing operation: Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from yolov8n_mct_model.onnx failed:Fatal error: Sony:MultiClassNMS(-1) is not a registered function/op

Reproduce with:

Download model here.

import mct_quantizers as mctq
import onnxruntime as ort

sess = ort.InferenceSession("yolov8n_mct_model.onnx",
                            mctq.get_ort_session_options(),
                            providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
Idan-BenAmi commented 2 weeks ago

Hi @ambitious-octopus , Can you try to add this additional import? from sony_custom_layers.pytorch.object_detection import nms_ort

ambitious-octopus commented 2 weeks ago

@Idan-BenAmi it works, thanks for that!