search

quic / ai-hub-models

The Qualcomm® AI Hub Models are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
https://aihub.qualcomm.com
BSD 3-Clause "New" or "Revised" License
338 stars 45 forks source link

[Feature Request] ControlNet app sample #44

Open Mengmengbai opened 1 month ago

Mengmengbai commented 1 month ago

Great works! Qualcomm has already supported ControlNet, but I have a confusion about how to run it on a mobile phone. As we know, there are some calculations between ControlNet and UNet, which are defined as an API by Qualcomm or do we need to implement this part ourselves? Or, is it necessary to convert both ControlNet and UNet into one model?

We hope to provide us with an Android sample for our development. We can't find relevant documents on the official website and other places.

mestrona-3 commented 1 month ago

Hi @Mengmengbai, I've added your request of a sample app to our backlog.

heydavid525 commented 1 month ago

You'd need to implement the lightweight "coordination code" in mobile native code (e.g., Java for Android, or javascript for electron app).

This demo submits optimized on-device binaries (text encoder, unet, vae decoder) and run inference on on device via AI Hub's inference job. The coordination code for these model components (e.g., DPM scheduler) are in pytorch / python. This demo gives you a sense of the numerics and performance for each model component, excluding the coordination code.

These binaries optimized for on-device can be used directly with good performance and numerics.