[Bug]: OpenVINOInferencer does not work with ONNX classification model

[haimat]

Describe the bug

I have trained a Fastflow model with task=TaskType.CLASSIFICATION for both the Folder and the Engine. After training I have exported the model to ExportType.ONNX. Now I want to load that model and predict images with it. This leads to the following error:

Traceback (most recent call last):
  File "/data/scratch/mkw-anomalib/anomalib-predict.py", line 35, in <module>
    result = inferencer.predict(image)
  File "/home/sinntelligence/.local/lib/python3.10/site-packages/anomalib/deploy/inferencers/openvino_inferencer.py", line 219, in predict
    output = self.post_process(predictions, metadata=metadata)
  File "/home/sinntelligence/.local/lib/python3.10/site-packages/anomalib/deploy/inferencers/openvino_inferencer.py", line 272, in post_process
    pred_score = anomaly_map.reshape(-1).max()
  File "/home/sinntelligence/.local/lib/python3.10/site-packages/numpy/core/_methods.py", line 41, in _amax
    return umr_maximum(a, axis, None, out, keepdims, initial, where)
ValueError: zero-size array to reduction operation maximum which has no identity

The problem lies in OpenVINOInferencer.post_process() (line 266ff). For some reason with the output from the ONNX model that if section ends up in the else part, where task = TaskType.SEGMENTATION. But this is wrong, because it's a classification model, as described above. Note that when I print the predictions after line 257 (predictions = predictions[self.output_blob]), then the output is an empty list: []. So apparently the ONNX model does not output correctly?

Dataset

Other (please specify in the text field below)

Model

FastFlow

Steps to reproduce the behavior

1. Install anomalib and train a (Fastflow) model with task=TaskType.CLASSIFICATION
2. Export that trained model via engine.export(model=model, export_type=ExportType.ONNX, export_root=...)
3. Create a second script with this code:

from anomalib.deploy.inferencers.openvino_inferencer import OpenVINOInferencer
from anomalib import TaskType
from PIL import Image

inferencer = OpenVINOInferencer(path="/path/to/ONNX/model.onnx", device="GPU", task=TaskType.CLASSIFICATION)
image = Image.open("/path/to/image.jpg")
result = inferencer.predict(image)

The last line leads to the aforementioned error.

OS information

OS information:

• OS: Ubuntu 22.04
• Python version: 3.10.12
• Anomalib version: 1.1,0
• PyTorch version: 2.2
• CUDA/cuDNN version: 12.2
• GPU models and configuration: 4x Nvidia RTX A6000
• Any other relevant information: I'm using a custom dataset

Expected behavior

I would expect the OpenVINOInferencer class works fine with ONNX classification models.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

(where do I find that?)

Logs

no logs

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[haimat]

Update: It seems to work when creating the OpenVINOInferencer with device="CPU". Does it not work with Nvidia GPUs?

[haimat]

Hey guys, any news on this? Or can you suggest any other way to run an exported anomlib model in ONNX format on the Nvidia GPU?

[ashwinvaidya17]

Hey @haimat, the team is quite busy at this moment, so we didn't have time to look into it. We will return to is as soon as possible. Meanwhile, you can take a look at the OpenVINO documentation to get an idea of the supported hardware, https://docs.openvino.ai/2024/home.html.

[haimat]

@ashwinvaidya17 Thanks for your response. I can see that Nvidia GPUs are not (fully) supported by OpenVINO, so I guess the best (= fastest) approach would be to convert the ONNX model into a TensorRT model. I guess that should be doable. But how can I read the output of the ONNX model, can I follow 1:1 what happens in the OpenVINOInferencer?

[haimat]

It seems the problem is related to Nvidia GPUs being not fully supported in OpenVINO. After I exported the model to ONNX / TensorRT format, everything works fine now.

[MMYY-yy]

After I exported the model to ONNX / TensorRT format, everything works fine now.

Hello, I also encountered this issue, but even after converting to onnx format, the GPU still doesn't work, I am looking forward to your answer very much output_path = Path("./results") openvino_model_path = output_path / "weights" / "onnx" / "model.onnx" metadata_path = output_path / "weights" / "onnx" / "metadata.json"

inferencer = OpenVINOInferencer(
    path=openvino_model_path,  # Path to the OpenVINO IR model.
    metadata=metadata_path,  # Path to the metadata file.
    task=TaskType.SEGMENTATION,
    device="GPU",  # We would like to run it on an Intel CPU.
)

The following error occurred after running：

RuntimeError: Exception from src/inference/src/cpp/infer_request.cpp:223: Check 'TRShape::merge_into(output_shape, in_copy)' failed at src/core/shape_inference/include\concat_shape_inference.hpp:49: While validating node 'opset1::Concat concat:/model/Concat_5 () -> ()' with friendly_name 'concat:/model/Concat_5': Shape inference input shapes {[0,64,64,64],[0,0,0,0],[0,0,0,0]} Argument shapes are inconsistent; they must have the same rank, and must have equal dimension everywhere except on the concatenation axis (axis 1).

BUT，when  device="AUTO"，Although the above error will not occur, the GPU is not used and the prediction time is very slow

[haimat]

I am not using the OpenVINOInferencer class for ONNX models. Instead I created a custom ONNX runner class - GPT & Co. are your friends :)

[MMYY-yy]

我没有将“OpenVINOInferencer”类用于 ONNX 模型。 相反，我创建了一个自定义 ONNX 运行器类 - GPT & Co. 是你的朋友 :)

Wow, your method is great. Would it be convenient to share it? I have encountered a big problem on my end

[haimat]

Well, first load the ONNX model:

import onnxruntime as ort

def load_onnx_model(self, path: str | Path) -> ort.InferenceSession:
    ort.set_default_logger_severity(3)
    providers = ["CUDAExecutionProvider"] if self.device == "CUDA" else ["CPUExecutionProvider"]
    sess_options = ort.SessionOptions()
    sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    return ort.InferenceSession(str(path), sess_options, providers=providers)

Then pre-process the image as desired (see OpenVINOInferencer), then call the model:

def forward_onnx(self, image: np.ndarray) -> np.ndarray:
    input_name = self.session.get_inputs()[0].name
    return self.session.run(None, {input_name: image})[0]

I have created most of the code via ChatGPT, it's straight forward :)

[MMYY-yy]

Well, first load the ONNX model:

import onnxruntime as ort

def load_onnx_model(self, path: str | Path) -> ort.InferenceSession:
    ort.set_default_logger_severity(3)
    providers = ["CUDAExecutionProvider"] if self.device == "CUDA" else ["CPUExecutionProvider"]
    sess_options = ort.SessionOptions()
    sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    return ort.InferenceSession(str(path), sess_options, providers=providers)

Then pre-process the image as desired (see OpenVINOInferencer), then call the model:

def forward_onnx(self, image: np.ndarray) -> np.ndarray:
    input_name = self.session.get_inputs()[0].name
    return self.session.run(None, {input_name: image})[0]

I have created most of the code via ChatGPT, it's straight forward :)

Your method looks great, thank you very much for sharing. I will try to modify the code, and if there are any parts that I don't know, I will ask you again. Thank you again for sharing.