[Bug]: OpenVINO crashes (SIGILL) when enumerating core.available_devices on J4105 and older

koush commented 2 months ago

OpenVINO Version

2024.3.0

Operating System

Other (Please specify in description)

Device used for inference

CPU

Framework

None

Model used

All Models

Issue description

OS: Ubuntu 22.04 LXC inside Proxmox.

OpenVINO crashes on J4105 and older (and possibly some generation newer) CPU. This issue started in 2024.3.0. Last good build was 2024.2.0 I can resolve the issue by simply downgrading the package.

~~Running this same code on the Proxmox host shell (instead of LXC) works fine. This may be a containerization issue.~~ See comment below where I've diagnosed the root cause.

Step-by-step reproduction

Repro code:

import openvino.runtime as ov

core = ov.Core()
print("Available devices:")
for device in core.available_devices:
    print(f"{device} :")

Relevant log output

(.venv) root@scrypted-wyse:/tmp/work# python3 test.py 
Available devices:
Illegal instruction

Issue submission checklist

[X] I'm reporting an issue. It's not a question.
[X] I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
[x] There is reproducer code and related data files such as images, videos, models, etc.

koush commented 2 months ago

The issue actually seems to stem from the apt package level-zero being installed. If I remove it, it no longer crashes. While this is not an issue with bare metal installation, it is an issue with creating docker images or lxc templates. The image can't contain the NPU support or risk crashing non-NPU systems.

level-zero is assuming that certain instructions exist while it is being probed.

rkazants commented 2 months ago

@ilya-lavrenov, could you please advice on this issue?

RyanMetcalfeInt8 commented 2 months ago

Just to add -- this issue has been reported several times for our updated set of of OpenVINO-based plugins for Audacity, which recently upgraded to use 2024.3. Here are a couple of examples:

https://forum.audacityteam.org/t/how-to-fix-unable-to-load-the-mod-openvino-module-error/116996/

https://github.com/intel/openvino-plugins-ai-audacity/issues/295

https://forum.audacityteam.org/t/cant-install-openvino-correctly-need-help-with-installation/118089/2

The workaround for these folks is to delete the NPU plugin DLL.

Edit: Note that the above reported issues have been on Windows, but seems to be the same issue (illegal instruction).

avitial commented 1 month ago

Ref. 154085

koush commented 2 weeks ago

@avitial has this been resolved in the latest level zero changes by chance? I noticed this in the changelog:

Remove preload of all drivers given zeInitDrivers and update unit tests

avitial commented 2 weeks ago

@koush yes, as I understand issue should have been addressed in the latest master branch, can you check in the latest openvino-nightly? I've tested on 2024.5.0.dev20241014 and Apollo Lake and the issue does not reproduce. Let us know your findings after testing.

koush commented 1 week ago

yes it seems resolved in the nightly

openvinotoolkit / openvino