search

quic / ai-hub-apps

The Qualcomm® AI Hub apps are a collection of state-of-the-art machine learning models optimized for performance (latency, memory etc.) and ready to deploy on Qualcomm® devices.
BSD 3-Clause "New" or "Revised" License
66 stars 15 forks source link

QCS6490 Support: Whisper, LLMs, and Android Package Availability #17

Closed yuguolong closed 5 days ago

yuguolong commented 1 week ago

This is a fantastic project, thank you for your hard work! I'm working with a Qualcomm Snapdragon QCS6490 chip running Android and have a few questions regarding its compatibility:

  1. Whisper model GPU support: Is the Whisper model capable of running on the QCS6490 GPU?
  2. LLM GPU support: Is it possible to run large language models on the QCS6490 GPU?
  3. Future Android package: Are there any plans to provide official Android packages for easier integration and deployment?

Thank you for your time and assistance.

kory commented 5 days ago

Whisper model GPU support: Is the Whisper model capable of running on the QCS6490 GPU?

Yes, you just need to select GPU when setting up TF Lite (the GPUv2 delegate). ONNX does not have a supported GPU path on the 6490: https://github.com/quic/ai-hub-apps/blob/ab1b5a7673803f3a6c99cc7a54f5c0c03af41624/apps/android/tflite_helpers/TFLiteHelpers.java#L49

LLM GPU support: Is it possible to run large language models on the QCS6490 GPU?

You'd likely run out of memory. We don't support LLMs on devices earlier than the 8 Gen 3 & 8 Elite.

Future Android package: Are there any plans to provide official Android packages for easier integration and deployment?

We offer QNN and the TFLite QNN delegate via Maven: https://github.com/quic/ai-hub-apps/blob/main/apps/android/ImageClassification/build.gradle#L58

We do not plan on distributing models themselves via an Android package, though.

yuguolong commented 4 days ago

@kory thank you for your reply.