search

android / ai-samples

Apache License 2.0
101 stars 14 forks source link

How does gemini nano work on qcom Npu/Gpu #7

Open AndreaChiChengdu opened 1 week ago

AndreaChiChengdu commented 1 week ago

I have a question. From the documentation and introduction, ai edge sdk api is used to access AICORE, while gemini nano is in AICORE. How does AICORE connect to Qualcomm npu hardware? Is it through runtime in AICORE or liteRT gpu delegate（tflite runtime） or Qualcomm's own qnn htp/gpu delegate? It's not mentioned in the document or the video. thanks

AndreaChiChengdu commented 1 week ago

@zhouyiself Is AICORE based on liteRT to connect to the Qualcomm acceleration library or is there another implementation in AICORE? Thank you very much https://github.com/android/ai-samples/issues/7#issue-2640088783

Flyfish233 commented 1 week ago

AICore on Galaxy device:

lib/arm64-v8a/libQnnHtp.so
lib/arm64-v8a/libQnnHtpV73Skel.so
lib/arm64-v8a/libQnnHtpV73Stub.so
lib/arm64-v8a/libQnnHtpV75Skel.so
lib/arm64-v8a/libQnnHtpV75Stub.so
lib/arm64-v8a/libQnnSystem.so
lib/arm64-v8a/libllm_qnn_jni.so

AICore thirdpartyeap:

lib/arm64-v8a/libllm_edgetpu_jni.so
lib/arm64-v8a/libllm_ggml_jni.so
lib/arm64-v8a/libulm1b_ggml_jni.so

That's enough to answer this issue.

AndreaChiChengdu commented 1 week ago

AICore on Galaxy device:

lib/arm64-v8a/libQnnHtp.so
lib/arm64-v8a/libQnnHtpV73Skel.so
lib/arm64-v8a/libQnnHtpV73Stub.so
lib/arm64-v8a/libQnnHtpV75Skel.so
lib/arm64-v8a/libQnnHtpV75Stub.so
lib/arm64-v8a/libQnnSystem.so
lib/arm64-v8a/libllm_qnn_jni.so

AICore thirdpartyeap:

lib/arm64-v8a/libllm_edgetpu_jni.so
lib/arm64-v8a/libllm_ggml_jni.so
lib/arm64-v8a/libulm1b_ggml_jni.so

That's enough to answer this issue.

Thanks a lot, but it looks like the pixel and sm8650 have different software paths? pixel uses ggml, Qualcomm npu as far as I know there are only two ways of tflite qnnhtp delegate and qnn offline compilation, and all signs show that aicore does not seem to use tflite runtime, which is where I do not understand. Thank you for your reply

Flyfish233 commented 1 week ago

AICore has several versions depending on the SoC, on Galaxy devices they may wrapped Qualcomm Genie SDK because that's the fastest API on a modern Qualcomm device with NPU, and they might not care about compabilities, so they probably won't use TFlite or GPUs.