search

stereolabs / zed-unity

ZED SDK Unity plugin
https://www.stereolabs.com/documentation/guides/using-zed-with-unity/introduction.html
MIT License
243 stars 57 forks source link

ZED Mini-Unity Object Detection error: Optimization process Timeout even after optimzing the AI models outside of Unity, using the ZED Diagnostic tool #269

Closed aaronlsmiles closed 9 months ago

aaronlsmiles commented 10 months ago

Preliminary Checks

Description

Optimization process times out even after optimzing the AI models outside of Unity, using the ZED Diagnostic tool, as instructed in the literature.

I have uninstalled and reinstalled, trying different versions of Unity (2021, 2022, etc.) and CUDA (11.8, 12.1, etc.), but the issue is always the same and the ZED Diagnostic tool shows everything with a green tick, except the AI Models, which show as unoptimised.

I also can't find the optimised AI models in any folder (e.g., C:\Program Files (x86)\ZED SDK, or C:\ProgramData\Stereolabs\resources (which is empty)).

Screenshot 2023-09-07 001423

Steps to Reproduce

I followed the steps for ZED 2 here, although I am using ZED Mini the documentation says object detection works with mini too.

  1. Plug in ZED Mini and load Unity
  2. Create a new scene and delete the Main Camera
  3. In the Project window, go to ZED -> Prefabs and drag ZED_Rig_Mono into the Hierarchy
  4. Select the new ZED_Rig_Mono in the Hierarchy
  5. In the Motion Tracking section, make sure Estimate Initial Position is checked. This enables floor detection
  6. Create a new empty GameObject in the Hierarchy and rename it “3D Object Visualizer”
  7. Add the ZED3DObjectVisualizer component to it
  8. In the Projects window, go to ZED -> Examples -> Object Detection -> Prefabs
  9. Drag the 3D Bounding Box prefab into Bounding Box Prefab in the Inspector

Expected Result

After a short initialization period, the app should pause for 10-20 seconds as the object detection module loads. Once it does, I should be able to step into the ZED’s view and see myself enclosed in a box or have objects that the model is trained on in the scene that 3D bounding boxes appear around.

Actual Result

Error: Optimization process Timeout. Please try to optimze the AI models outside of Unity, using the ZED Diagnostic tool UnityEngine.Debug:LogError (object) ZEDManager/d__379:MoveNext () (at Assets/ZED/SDK/Helpers/Scripts/ZEDManager.cs:3037) UnityEngine.SetupCoroutine:InvokeMoveNext (System.Collections.IEnumerator,intptr)

ZED Camera model

ZED Mini

Environment

Edition Windows 11 Home Version 22H2 Installed on ‎28/‎07/‎2023 OS build 22621.2134 Experience Windows Feature Experience Pack 1000.22659.1000.0 Processor 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz 2.30 GHz Installed RAM 32.0 GB (31.7 GB usable) Device ID 89E4F234-8C60-4866-B422-24FBAD207069 Product ID 00326-10000-00000-AA011 System type 64-bit operating system, x64-based processor

ZED_Diagnostic_Results:

{
    "AI Models": {
        "info": [
            "MULTI CLASS DETECTION is not optimized",
            "MULTI CLASS MEDIUM DETECTION is not optimized",
            "MULTI CLASS ACCURATE DETECTION is not optimized",
            "HUMAN BODY FAST DETECTION is not optimized",
            "HUMAN BODY MEDIUM DETECTION is not optimized",
            "HUMAN BODY ACCURATE DETECTION is not optimized",
            "HUMAN BODY 38 FAST DETECTION is not optimized",
            "HUMAN BODY 38 MEDIUM DETECTION is not optimized",
            "HUMAN BODY 38 ACCURATE DETECTION is not optimized",
            "PERSON HEAD DETECTION is not optimized",
            "PERSON HEAD ACCURATE DETECTION is not optimized",
            "REID ASSOCIATION is not optimized",
            "NEURAL DEPTH is not optimized"
        ]
    },
    "Camera Test": {
        "InternalDevicesCount": 2,
        "ZEDCount": 1,
        "camera": {
            "Device ID": 1,
            "Firmware": 1523,
            "Init Output": "OK",
            "Model": "ZED Mini",
            "Serial Number": 16396036,
            "valid": [
                "<b>Camera: </b>ZED Mini",
                "<b>Firmware: </b>1523"
            ]
        },
        "resolutions": {
            "HD1080@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD1080@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 1080,
                "height (output)": 1080,
                "initialization": "OK",
                "status": true,
                "width (input)": 1920,
                "width (output)": 1920
            },
            "HD2K@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 1242,
                "height (output)": 1242,
                "initialization": "OK",
                "status": true,
                "width (input)": 2208,
                "width (output)": 2208
            },
            "HD720@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 720,
                "height (output)": 720,
                "initialization": "OK",
                "status": true,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "HD720@60": {
                "dropCount": 0,
                "fps (input)": 60,
                "fps (output)": 59,
                "height (input)": 720,
                "height (output)": 720,
                "imageCount": 501,
                "initialization": "OK",
                "status": true,
                "tearingCount": 0,
                "width (input)": 1280,
                "width (output)": 1280
            },
            "VGA@100": {
                "fps (input)": 100,
                "fps (output)": 91,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@15": {
                "fps (input)": 15,
                "fps (output)": 15,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@30": {
                "fps (input)": 30,
                "fps (output)": 30,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            },
            "VGA@60": {
                "fps (input)": 60,
                "fps (output)": 47,
                "height (input)": 376,
                "height (output)": 376,
                "initialization": "OK",
                "status": true,
                "width (input)": 672,
                "width (output)": 672
            }
        },
        "sensors": {
            "Firmware": 517,
            "Init Output": "OK",
            "Model": "ZED Mini (MCU)",
            "Serial Number": 16396036
        }
    },
    "Devices": {
        "ASMediaFound": false,
        "CorruptedFirmware": false,
        "MCUDetected": true,
        "OC_Cam_Detected": false,
        "OtherUSBDevices": [
            {
                "USBMode": 3,
                "USB_path": "/19",
                "bDescriptorType": 1,
                "bDeviceProtocol": 0,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "2.4",
                "bcdUSB": "3.0",
                "bcdUSBClass": 0,
                "bcdUSBSubClass": 0,
                "busNumber": 2,
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 3,
                "idProduct": 790,
                "idVendor": 3034
            }
        ],
        "USBList": [
            {
                "USB_path": "/12",
                "idProduct": "0x6005",
                "idVendor": "0x048d"
            },
            {
                "USB_path": "/12",
                "idProduct": "0x6005",
                "idVendor": "0x048d"
            },
            {
                "USB_path": "/12",
                "idProduct": "0x6005",
                "idVendor": "0x048d"
            },
            {
                "USBMode": 3,
                "USB_path": "/17",
                "bDescriptorType": 1,
                "bDeviceProtocol": 1,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "1.0",
                "bcdUSB": "3.0",
                "bcdUSBClass": 239,
                "bcdUSBSubClass": 2,
                "busNumber": 2,
                "device": "ZED mini",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 0,
                "idProduct": "0xf682",
                "idVendor": "0x2b03"
            },
            {
                "USBMode": 2,
                "USB_path": "/1",
                "bDescriptorType": 1,
                "bDeviceProtocol": 0,
                "bLength": 18,
                "bMaxPacketSize0": 64,
                "bNumConfigurations": 1,
                "bcdDevice": "2.5",
                "bcdUSB": "2.0",
                "bcdUSBClass": 0,
                "bcdUSBSubClass": 0,
                "busNumber": 2,
                "device": "ZED mini MCU",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 3,
                "idProduct": "0xf681",
                "idVendor": "0x2b03"
            },
            {
                "USBMode": 3,
                "USB_path": "/17",
                "bDescriptorType": 1,
                "bDeviceProtocol": 1,
                "bLength": 18,
                "bMaxPacketSize0": 9,
                "bNumConfigurations": 1,
                "bcdDevice": "1.0",
                "bcdUSB": "3.0",
                "bcdUSBClass": 239,
                "bcdUSBSubClass": 2,
                "busNumber": 2,
                "device": "ZED mini",
                "iManufacturer": 1,
                "iProduct": 2,
                "iSerial": 0,
                "idProduct": "0xf682",
                "idVendor": "0x2b03"
            },
            {
                "USB_path": "/19",
                "idProduct": "0x0316",
                "idVendor": "0x0bda"
            },
            {
                "USB_path": "/11",
                "idProduct": "0xce00",
                "idVendor": "0x048d"
            },
            {
                "USB_path": "/4",
                "idProduct": "0xc52b",
                "idVendor": "0x046d"
            },
            {
                "USB_path": "/4",
                "idProduct": "0xc52b",
                "idVendor": "0x046d"
            },
            {
                "USB_path": "/14",
                "idProduct": "0x0029",
                "idVendor": "0x8087"
            },
            {
                "USB_path": "/13",
                "idProduct": "0xb71a",
                "idVendor": "0x04f2"
            },
            {
                "USB_path": "/13",
                "idProduct": "0xb71a",
                "idVendor": "0x04f2"
            }
        ],
        "USBMode": 3,
        "ZED Camera Module Detected": 63106,
        "ZED MCU Module Detected": 63105,
        "ZEDDetected": true,
        "captureDevices": [
        ],
        "controllers": [
            {
                "CompatibleIDs": "PCI\\VEN_8086&DEV_43ED&REV_11",
                "Description": "USB xHCI Compliant Host Controller",
                "DeviceName": "Intel(R) USB 3.20 eXtensible Host Controller - 1.20 (Microsoft)",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0001",
                "Location": "PCI bus 0, device 20, function 0",
                "Manufacturer": "Generic USB xHCI Host Controller",
                "Name": "PCI",
                "PhysicalName": "\\Device\\NTPNP_PCI0009"
            },
            {
                "CompatibleIDs": "",
                "Description": "USB Root Hub (USB 3.0)",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0003",
                "Location": "",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-0"
            },
            {
                "CompatibleIDs": "PCI\\VEN_8086&DEV_9A17&REV_05",
                "Description": "USB xHCI Compliant Host Controller",
                "DeviceName": "Intel(R) USB 3.10 eXtensible Host Controller - 1.20 (Microsoft)",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0000",
                "Location": "PCI bus 0, device 13, function 0",
                "Manufacturer": "Generic USB xHCI Host Controller",
                "Name": "PCI",
                "PhysicalName": "\\Device\\NTPNP_PCI0007"
            },
            {
                "CompatibleIDs": "",
                "Description": "USB Root Hub (USB 3.0)",
                "DeviceName": "",
                "Driver": "{36fc9e60-c465-11cf-8056-444553540000}\\0002",
                "Location": "",
                "Manufacturer": "(Standard USB HUBs)",
                "Name": "USB",
                "PhysicalName": "\\Device\\USBPDO-1"
            }
        ],
        "valid": [
            "<b>USB Bandwidth: </b> OK"
        ]
    },
    "Graphics Card": {
        "deviceCount": 1,
        "deviceDriverVersion": 12010,
        "devices": [
            {
                "arch": "Ampere",
                "computeCapability": "8.6",
                "cores": 6144,
                "name": "NVIDIA GeForce RTX 3080 Laptop GPU",
                "totalMemoryBytes": "17179344896",
                "totalMemoryMB": 16383.5
            }
        ],
        "initResult": 0,
        "valid": [
            "<b>Graphics card: </b> NVIDIA GeForce RTX 3080 Laptop GPU"
        ]
    },
    "Processor": {
        "OS": "Windows 10 FallCreators",
        "avxSupported": true,
        "coreCount": "8",
        "cpu": "11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
        "datetime": "2023-09-07 00:11:51",
        "hyperThreading": "1",
        "motherboard": "GM5TN8Y, TongFang",
        "ramCapacity": "31.7471",
        "ramUsage": "0",
        "ramUsed": "16.697",
        "threadCount": "16",
        "valid": [
            "<b>Processor: </b> 11th Gen Intel(R) Core(TM) i7-11800H @ 2.30GHz",
            "<b>Motherboard: </b> GM5TN8Y, TongFang"
        ]
    },
    "ZED SDK": {
        "CUDA Toolkit version": "V12.1.66 -->located in C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/bin/nvcc.exe",
        "CUDA loading": false,
        "ZED SDK Version (Diag)": "4.0.6",
        "ZED SDK Version (RT)": "4.0.6",
        "ai": {
            "checkAI": true,
            "cudaVersion": 12010,
            "cudnnCudartVersion": 12000,
            "cudnnVersion": 8801,
            "cudnnVersionExpected": 8801,
            "tensorVersion": 8600
        },
        "binFiles": [
            "cudnn64_8.dll",
            "cudnn_adv_infer64_8.dll",
            "cudnn_adv_train64_8.dll",
            "cudnn_cnn_infer64_8.dll",
            "cudnn_cnn_train64_8.dll",
            "cudnn_ops_infer64_8.dll",
            "cudnn_ops_train64_8.dll",
            "nvinfer.dll",
            "nvinfer_builder_resource.dll",
            "nvinfer_dispatch.dll",
            "nvinfer_lean.dll",
            "nvinfer_plugin.dll",
            "nvinfer_vc_plugin.dll",
            "nvonnxparser.dll",
            "nvparsers.dll",
            "sl_ai64.dll",
            "sl_zed64.dll",
            "zlibwapi.dll"
        ],
        "driverVersion": "531.61",
        "drivervDllVersion": "31.0.15.3161",
        "resourcesFiles": [
        ],
        "resourcesFilesExpected": [
            "objects_performance_3.2"
        ],
        "runtimeDllVersion": 12010,
        "valid": [
            "<b>ZED SDK version:</b> 4.0.6",
            "<b>CUDA version:</b> V12.1.66"
        ]
    }
}

Anything else?

No response

aaronlsmiles commented 9 months ago

This should be possible with ZED Mini as it states here:

Important: At the moment, only a few object classes can be detected and tracked with the 3D Object Detection API using ZED cameras (except the ZED 1 camera).

And here:

Note: Object Detection has been optimized for ZED 2/ZED 2i and uses the camera motion sensors for improved reliability. Therefore the Object Detection module requires a ZED 2/ZED 2i or ZED Mini, and sensors cannot be disabled when using the module.

But there the tutorial only addresses ZED 2.

Can anyone elaborate please, this is urgent!?

SLJLMacarit commented 9 months ago

Hi @aaronlsmiles, sorry for the delay. You should very much be able to optimize the models, at least outside of Unity. Does optimizing the models outside of Unity work? You can find some directives here: How can I optimize the ZED SDK AI models manually?

aaronlsmiles commented 9 months ago

I contacted support and there was some issue with the optimised files not downloading as the resources folder was always empty. So they sent them across and I pasted them in the folder then optimised them, which worked. They are investigating the issue.