search

quic / ai-hub-models

The Qualcomm® AI Hub Models are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
https://aihub.qualcomm.com
BSD 3-Clause "New" or "Revised" License
419 stars 58 forks source link

[BUG] IOT BYOM Issue: Compiling PyTorch model to a QNN Context Binary on AiHub fails #95

Open Midi12 opened 2 weeks ago

Midi12 commented 2 weeks ago

Describe the issue Please provide details relating to the issue you're hitting, if it is related to performance, accuracy or other model issues with bringing your own model to Qualcomm AI Hub, to deploy to IOT devices.

Hello, firstly thanks for the wonderful AiHub Platform :)

I am trying some example

To Reproduce Below the minimal example :

  1. Run this minimal example 
    
import qai_hub as hub
import torch
from torchvision.models import mobilenet_v2
import requests
import numpy as np
from PIL import Image

Using pre-trained MobileNet

torch_model = mobilenet_v2(pretrained=True) torch_model.eval()

Step 1: Trace model

input_shape = (1, 3, 224, 224) example_input = torch.rand(input_shape) traced_torch_model = torch.jit.trace(torch_model, example_input)

Step 2: Compile model

compile_job = hub.submit_compile_job( model=traced_torch_model, device=hub.Device("QCS6490 (Proxy)"), options="--target_runtime qnn_context_binary", input_specs=dict(image=input_shape), )

2. Go to the compile job url to assess it failed
3. Open the compile log

It fails with this error

```2024-08-27 07:19:36,717 -   ERROR - qnn-model-lib-generator: stderr : jni/tmp23ncqrsx.cpp:38:51: warning: mixture of designated and non-designated initializers in the same initializer list is a C99 extension [-Wc99-designator]
                                                  .hybridCoo= {.numSpecifiedElements= 0, .numSparseDimensions= 0}},
                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expected behavior The model is compiled to a QNN context binary.

Compilation logs link1 link2

Host configuration:

Additional context Add any other context about the problem here.

heydavid525 commented 2 weeks ago

I can take a look at it. Can you share the link to the failed compile job? Thanks!

Midi12 commented 2 weeks ago

I can take a look at it. Can you share the link to the failed compile job? Thanks!

Below are the related links, thanks.

https://app.aihub.qualcomm.com/jobs/jqpyy498p/ https://app.aihub.qualcomm.com/jobs/j0pxz7y85/

heydavid525 commented 1 week ago

Hi @Midi12 , QCS6490 is RB-Gen2, which supports int8 only. I tried the same script but changed to the following options for compilation, it compiles successfully.

--target_runtime qnn_context_binary --quantize_full_type=int8 --quantize_io

To help quantization numerics, you would need to supply calibration_data` to help with the quantization.

Hope this helps!

Midi12 commented 5 days ago

Hello @heydavid525 , yes this resolve my issue, thanks !

Midi12 commented 3 days ago

Hello, I am reopening this issue as I encountered the same error message with a larger model (Whisper medium from here https://github.com/openai/whisper/blob/main/whisper/__init__.py#L17).

Here is the links to related jobs https://app.aihub.qualcomm.com/jobs/j2p0w2425/ https://app.aihub.qualcomm.com/jobs/j7gj9z9ep/

One of two jobs is failing.